Commit 1fad4d65 authored by Nicolas Denoyelle's avatar Nicolas Denoyelle Committed by Swann Perarnau
Browse files

[doc] Add doxygen documentation to the project

parent 10a2b644
......@@ -3,6 +3,7 @@
# unignore files with extensions
!*.*
# unignore special files
!AUTHORS
!LICENSE
......@@ -50,7 +51,7 @@ stamp-h1
/install-sh
/missing
/stamp-h1
version.h
/version.h
/m4/libtool.m4
/m4/ltmain.sh
/m4/ltoptions.m4
......
......@@ -4,6 +4,7 @@ variables:
stages:
- style
- build
- release
make:generic:
stage: build
......@@ -53,3 +54,12 @@ checkpatch:
stage: style
script:
- nix run -f "$ARGOPKGS" checkpatch --command checkpatch.pl
readthedocs:
stage: release
when: on_success
only:
- master
- /v\.[0-9]+\.[0-9]+\.[0-9]+/
script:
- nix run nixpkgs.curl -c curl -X POST -d "branch=$CI_COMMIT_REF_NAME" -d "token=$READTHEDOCS_TOKEN" https://readthedocs.org/api/v2/webhook/argo-aml/83161/
Contributing to The Project
===========================
AML Contribution Guide {#contributing_page}
============
The following is a set of guidelines for contributing to this project. These
are guidelines, not rules, so use your best judgment and feel free to propose
changes. For information about the overall design of the library and general
coding guidelines, see the `DESIGN.markdown` file.
Welcome to AML contribution guide.
This page walk through the main library expectations.
When contributing to AML, contributors should write
features aligned with AML goal and AML spirit.
Additionnally, several good practices are requested, some beeing inforced
through the continuous integration (CI) pipeline.
For readability and maintainability, coding conventions need to be respected,
code needs to be tested, documented and integrated with the main AML branch
and its build tool-chain.
## Contribution Workflow
## General Guidelines
At its core, AML is able to provide access to established, well known
abstractions inside a common toolbox, with the more flexibility possible.
Abstractions are made available through __building blocks__, each representing
a single abstraction. Each building block is separated into a __generic__ API
for operations that the abstraction provides, and __implementations__ of those
operations for a specific implementation of the abstraction.
__generic__ API, i.e the first layer exposed to users
through a generic interface declares a table of operations common to the
abstraction, the main abstraction structure containing a such a table
and an opaque handle to store implementation specific data.
```
struct aml_building_block_ops{
aml_building_block_op0();
...
};
struct aml_building_block_data;
struct aml_building_block{
struct aml_building_block_ops *ops;
struct aml_building_block_data *data;
};
```
A set of generic functions taking the abstraction as first argument
is supposed to be exposed while it will de-reference underlying
implementation function table to implement expected behavior.
Such generic functions are supposed to return an AML error code as defined
in [error header](@ref error.h).
```
int aml_building_block_feature(struct aml_building_block *, ...);
```
Building blocks should interact with each other through the generic API, but
specific implementations are allowed to break this rule for
performance-oriented reasons. A generic version should still be available.
It is expected that these will be transparent and composable,
because low-level optimizations require control.
Therefore, most implementation details (structures, functions) will be exposed
to the user in a dedicated header, and use of opaque structures, static functions
and variables there should be avoided.
Moreover, customization of a building block should happen at creation time: the goal
is to make it easy for users to evaluate different configurations options with
minimal impact on the inner code of the application.
Therefore, building blocks specialized implementation, must declare a dynamic allocator, free,
initialization and finish functions, and stack declaration.
```
int aml_building_block_create (struct aml_building_block **, ...);
void aml_building_block_destroy(struct aml_building_block *);
int aml_building_block_init (struct aml_building_block *, ...);
void aml_building_block_fini (struct aml_building_block *);
#define AML_BUILDING_BLOCK_DECL(name) \
struct aml_building_block_data __ ##name## _inner_data; \
struct aml_building_block name = { \
&aml_building_block_ops, \
(struct aml_building_block_data *)&__ ## name ## _inner_data, \
}
```
Usually a building blocks implementation would declare an exportable (`extern`)
`struct aml_building_block_ops`, and `struct aml_building_block`, to provide
users with a default static implementation of the block.
Finally, building blocks headers, sources and tests go to a dedicated folder
located respectively in `include`, `src` and `tests`.
## Coding Convention
AML code respects several coding conventions. Some of them are enforced through
the CI pipeline.
Prior to start coding AML, you might want to set your editor to minimize
the process of code formatting.
Most of the formatting can also be scripted with `indent` program.
Code formatting compliance with AML requirements can be checked by running
[linux kernel linter](https://github.com/torvalds/linux/blob/master/scripts/checkpatch.pl)
at the root of AML project.
The list of coding convention is the following:
* use tabulation of 8 characters,
* remove trailing white spaces,
* lines may not exceed 80 characters,
* pointers star have to stick to the variable name: `void *var;`,
* function return type is on the same line as function name,
* curly braces are on the same line as the statement or function,
* A new line must appear after curly braces.
* macros are UPPERCASE_WITH_UNDERSCORES, function lowercase_with_underscores
* All functions/macros should start with aml/AML followed by the name of the building block involved.
## Versioning and Submission Workflow
AML code and CI runners is hosted on Argonne National Laboratory
[gitlab](https://xgitlab.cels.anl.gov/argo/aml) infrastructure under git version control.
New user may create new branches upon request to host and test their code.
New branched must be created from the master branch and merging is done
via git or gitlab pull request interface.
We basically follow the [Gitlab Flow](https://docs.gitlab.com/ee/workflow/gitlab_flow.html)
model:
- `master` is the primary branch for up-to-date development
- any new feature/development should go into a `feature-branch` based of
- any new feature/development should go into a `feature-branch` based on
`master` and merge into `master` after.
- we follow a `semi-linear history with merge commits` strategy: each merge
request should include clean commits, but only the HEAD and merge commit
......@@ -21,7 +126,8 @@ model:
change should appear in its own commit.
- to help with in-development work, CI will not activate on branches with
names started with wip/
## Commit Messages Styleguide
### Commit Messages Styleguide
- use present tense, imperative mood
- reference issues and merge requests
......@@ -59,7 +165,7 @@ commit.template my-config-template.txt` to use it automatically.
# --------------------
```
## Signoff on Contributions:
### Signoff on Contributions:
The project uses the [Developer Certificate of
Origin](https://developercertificate.org/) for copyright and license
......@@ -69,3 +175,98 @@ the `LICENSE` file) and that you agree to the DCO.
To signoff commits, use: `git commit --signoff`.
To signoff a branch after the fact: `git rebase --signoff`
## Testing
AML project includes a set of unit and integration tests.
Each AML building block has its own set of tests in a separate directory.
Tests are launched together with automake test suite with the `make check` command.
Building blocks should be tested as thoroughly as possible and whenever
relevant, integration tests with other components should be available.
Tests compilation and launch are implemented together in `tests/Makefile.am`.
Whenever a branch is pushed to AML repository, it automatically triggers
the CI pipeline, unless branch name starts with `wip/`.
The CI pipeline must be successful for a merge request to be accepted.
## Documentation
For obvious reasons, the code and building block must be documented.
The whole documentatation is based doxygen tool.
All the public code (data structures, functions, macro, variables) must be
documented as precisely as possible. Additionnaly, new buiding blocks implementation
must create a group (with `\@defgroup`) and provide a detailed description of
the features it provides. Syntax for code documentation can be found
[here](http://www.doxygen.nl/manual/docblocks.html),
and most useful elements can be found in the example below.
```
/**
* \@defgroup aml_buidling_block_implementation "AML buidling block implemation"
*
* \@brief This is a short description of this group.
*
* The documentation encapsulated below
* including this long description of the building_block
* implementation will be create a section dedicated to this buidling block
* implementation. It will be possible to link to it in the main header
* documentation with "@ref" or "@see".
* \@{
**/
/**
* This is a default operation table implementing
* table defined in main header.
**/
extern struct aml_building_block_ops aml_building_block_ops;
/**
* This is the bb inner data declaration, to
* stuff with needed attributes.
**/
struct aml_building_block_data {};
/**
* This is the bb declaration
**/
struct aml_building_block{
struct aml_building_block_ops *ops;
struct aml_building_block_data *data;
};
/** This is the static buidling block declaration macro **/
#define AML_BUILDING_BLOCK_DECL(name) \
struct aml_building_block_data __ ##name## _inner_data; \
struct aml_building_block name = { \
&aml_building_block_ops, \
(struct aml_building_block_data *)&__ ## name ## _inner_data, \
}
/** Static declaration of the building block size **/
#define AML_BUILDING_BLOCK_ALLOCSIZE (sizeof(struct aml_building_block_data) + \
sizeof(struct aml_building_block))
/**
* Allocates and initializes a new building_block.
*
* \@param bb[out]: A pointer to store the new building block.
* \@param other[in]: Other parameters used as input.
* \@return AML_SUCCESS if successful; an error code otherwise.
**/
int aml_building_block_create(struct aml_building_block **bb, ...);
...
/**
* This is the end of the inclusion into building_block documentation group.
* \@}
**/
```
If the contribution is an implementation of a building block and header has
been placed in `include/aml/building_block` then its documentation will automatically added to
the project documentation. However, it is necessary to point to this particular group
with `@ref` or `@see` command in the [main aml header](\ref aml.h).
If the contribution creates new include directories, the latter have to be included
in `doc/aml.doxy` after the field `INPUT`.
......@@ -5,7 +5,12 @@ if ADD_BENCHMARKS
SUBDIRS += benchmarks
endif
if AML_DOXYGEN
SUBDIRS += doc
endif
pkgconfigdir = $(libdir)/pkgconfig
pkgconfig_DATA = aml.pc
EXTRA_DIST = autogen.sh aml.pc README.markdown
......@@ -7,6 +7,30 @@ memory management policies for allocation and placement of data across devices.
This library is still in the prototyping phase. APIs might break often.
## Documentation
The latest documentation can be found
[online](https://argo-aml.readthedocs.io/en/latest/),
can be installed when installing a release or generated
from git clone before installing with
```
make -C doc doxygen
```
## Requirements:
* autoconf
* automake
* libnuma
## Installation
```
sh autogen.sh
./configure
make -j install
```
# General Architecture
The architecture of the library relies on two principles: type-specific
......
......@@ -69,6 +69,12 @@ AC_CHECK_HEADERS([numa.h],,[AC_MSG_ERROR([AML requires libnuma headers.])])
AC_CHECK_HEADERS([numaif.h],,[AC_MSG_ERROR([AML requires libnuma headers.])])
AC_CHECK_LIB(numa, mbind,,[AC_MSG_ERROR([AML requires libnuma.])])
# Doxygen
# See m4/ax_doxygen.m4
AC_CHECK_PROG(doxygen_happy, [doxygen], [yes], [no])
AM_CONDITIONAL([AML_DOXYGEN],[ test "x$doxygen_happy" = "xyes" ])
AC_CONFIG_HEADERS([include/config.h])
AC_CONFIG_FILES([Makefile
......@@ -80,6 +86,7 @@ AC_CONFIG_FILES([Makefile
src/utils/Makefile
include/Makefile
tests/Makefile
doc/Makefile
benchmarks/Makefile
aml.pc
include/aml/utils/version.h])
......
DATA_INSTALL_DIR=$(datadir)/aml
DOXYGEN_BUILD_DIR=./build-doxygen
SPHINX_BUILD_DIR=./build-sphinx
doxygen:
doxygen aml.doxy
create-doc-dir:
mkdir -p $(DATA_INSTALL_DIR)
install-doc-html:
if [ -d $(DOXYGEN_BUILD_DIR)/html ]; then \
cp -r $(DOXYGEN_BUILD_DIR)/html $(DATA_INSTALL_DIR); \
fi
install-data-local: create-doc-dir install-doc-html
uninstall-local:
rm -rf $(DATA_INSTALL_DIR)
clean-local:
rm -rf $(DOXYGEN_BUILD_DIR) $(SPHINX_BUILD_DIR)
dist-hook: doxygen
cp -r $(DOXYGEN_BUILD_DIR) $(distdir)
This diff is collapsed.
################################################################################
# Copyright 2019 UChicago Argonne, LLC.
# (c.f. AUTHORS, LICENSE)
#
# This file is part of the AML project.
# For more info, see https://xgitlab.cels.anl.gov/argo/aml
#
# SPDX-License-Identifier: BSD-3-Clause
################################################################################
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# http://www.sphinx-doc.org/en/master/config
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
import subprocess
# Generate documentation
subprocess.call('doxygen aml.doxy', shell=True)
# -- General configuration ---------------------------------------------------
# The language for content autogenerated by Sphinx.
language = 'C'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = []
# Add any paths that contain templates here, relative to this directory.
templates_path = []
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []
# -- Project information -----------------------------------------------------
project = 'AML'
copyright = 'Copyright 2019 UChicago Argonne, LLC'
author = 'Swann Perarnau, Kamil Iskra, Brian Suchy, Valentin Reis, Nicolas Denoyelle'
release = 'X.X.X'
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = []
#Overwrite sphinx documentation with doxygen documentation.
html_extra_path = ['build-doxygen/html']
This is an empty file required by shpinx when
compiling documentation on readthedocs.org.
This is an empty landing page overwritten with doxygen documentation.
Welcome to AML Documentation Page {#index}
============
AML is a memory management library designed for highly optimized
codes to delegate complex memory operations.
## Overview
AML goals are the following:
1. to provide a convenient interface for addressing all heterogeneous
addressable memories and moving data around.
2. to provide an interface for expressing and translating data
structures between several representations and physical memories.
Toward achieving these goals, AML implements the following abstractions:
\image html building-blocks-diagram.png width=400cm
* [Areas](@ref aml_area), a set of addressable physical memories,
* [Tilings](@ref aml_tiling), a description of tiling data structures,
* [DMA](@ref aml_dma), an engine to asynchronously move data structures between areas,
* [Scratchpad](@ref aml_scratch), a stage-in, stage-out abstraction for prefetching.
Each of these abstractions have several implementations. For instance,
areas may refer to usual DDR of a subset of them, GPU memory or non-volatile memory.
Tilings are implemented to reflect either 1D or 2D structures etc.
## Problems Solved with AML
AML library is designed for highly optimized codes to delegate complex memory
operations. AML is currently used in a
[proof of concept](https://xgitlab.cels.anl.gov/argo/nauts/tree/master/papers/mchpc18)
of efficient matrix multiplication on architectures with software managed memory side
cache. It is also currently being extended toward specialized memory allocations
for latency sensitive applications.
## Quick Start Guide
### Download
* [Github version](https://xgitlab.cels.anl.gov/argo/aml):
```
git clone git@xgitlab.cels.anl.gov:argo/aml.git
```
* Release versions:
TODO after release.
### Requirements:
* autoconf
* automake
* libnuma
### Installation
```
sh autogen.sh
./configure
make -j install
```
### Workflow
* Include aml header:
```
#include <aml.h>
...
int main(int argc, char **argv){
```
* Check AML version:
```
if(aml_version_major != AML_VERSION_MAJOR){
printf("AML ABI mismatch!");
return 1;
}
```
* Initialize and Cleanup AML:
```
if(aml_init(&argc, &argv) != 0){
printf("AML library init failure!");
return 1;
}
...
aml_finalize();
```
* Link your program with `-laml`
See above building blocks specific pages for further examples and information on
library features.
## Support
AML interface with users is mainly done through
[gitlab issue interface](https://xgitlab.cels.anl.gov/argo/aml/issues) and
direct contact with developers/maintainers:
* Swann Perarnau (swann@anl.gov)
* Nicolas Denoyelle (ndenoyelle@anl.gov)
AML is looking forward to new users. If you think your problem can be solved with AML
eventually with some additional non-existing features which are relevant to the library,
please let us know. Tell us about your topic of interests and your scientific/technical
problems. We are also pleased to improve the software, thus any suggestion, bug
declaration, contribution is welcomed.
## Contributing
AML is looking forward to new collaborations and contributions.
If you wish to contribute to AML project rendez-vous [here](@ref contributing_page)
This diff is collapsed.
......@@ -15,81 +15,86 @@
#include <numa.h>
#include <numaif.h>
/* allowed binding flags */
/**
* @defgroup aml_area_linux "AML Linux Areas"
* @brief Linux Implementation of Areas.
*
* Linux implementation of AML areas.
* This building block relies on libnuma implementation and
* linux mmap/munmap to provide mmap/munmap on NUMA host
* host processor memory. New areas may be created
* to allocate a specific subset of memories.
* This building block also include a static declaration of
* a default initialized area that can be used out of the box with
* abstract area API.
*
* #include <aml/area/linux.h>
* @{
**/
/**
* Allowed binding flag for area creation.
* This flag will apply strict binding to the selected bitmask.
* If subsequent allocation will failt if they cannot enforce binding
* on bitmask.
**/
#define AML_AREA_LINUX_BINDING_FLAG_BIND (MPOL_BIND)
/**
* Allowed binding flag for area creation.
* This flag will make subsequent allocations to interleave
* pages on memories of the bitmask.
**/
#define AML_AREA_LINUX_BINDING_FLAG_INTERLEAVE (MPOL_INTERLEAVE)
/**
* Allowed binding flag for area creation.
* This flag will make subsequent allocations to bound to the
* nodes of bitmask if possible, else to some other node.
**/
#define AML_AREA_LINUX_BINDING_FLAG_PREFERRED (MPOL_PREFERRED)
/* allowed mmap flags to pass */
/**
* Allowed mapping flag for area creation.
* This flag will make subsequent allocations to be private
* to the process making them.
**/
#define AML_AREA_LINUX_MMAP_FLAG_PRIVATE (MAP_PRIVATE | MAP_ANONYMOUS)
#define AML_AREA_LINUX_MMAP_FLAG_SHARED (MAP_SHARED | MAP_ANONYMOUS)
extern struct aml_area_ops aml_area_linux_ops;
/* User data stored inside area */
struct aml_area_linux_data {
/** numanodes to use **/
struct bitmask *nodeset;
/** numaif.h mbind policy or AML_AREA_LINUX_FLAG_* **/
int binding_flags;
/** mmap flags **/
int mmap_flags;
};
/* Default linux area with private mapping and no binding. */