Commit 1fad4d65 authored by Nicolas Denoyelle's avatar Nicolas Denoyelle Committed by Swann Perarnau

[doc] Add doxygen documentation to the project

parent 10a2b644
......@@ -3,6 +3,7 @@
# unignore files with extensions
!*.*
# unignore special files
!AUTHORS
!LICENSE
......@@ -50,7 +51,7 @@ stamp-h1
/install-sh
/missing
/stamp-h1
version.h
/version.h
/m4/libtool.m4
/m4/ltmain.sh
/m4/ltoptions.m4
......
......@@ -4,6 +4,7 @@ variables:
stages:
- style
- build
- release
make:generic:
stage: build
......@@ -53,3 +54,12 @@ checkpatch:
stage: style
script:
- nix run -f "$ARGOPKGS" checkpatch --command checkpatch.pl
readthedocs:
stage: release
when: on_success
only:
- master
- /v\.[0-9]+\.[0-9]+\.[0-9]+/
script:
- nix run nixpkgs.curl -c curl -X POST -d "branch=$CI_COMMIT_REF_NAME" -d "token=$READTHEDOCS_TOKEN" https://readthedocs.org/api/v2/webhook/argo-aml/83161/
Contributing to The Project
===========================
AML Contribution Guide {#contributing_page}
============
The following is a set of guidelines for contributing to this project. These
are guidelines, not rules, so use your best judgment and feel free to propose
changes. For information about the overall design of the library and general
coding guidelines, see the `DESIGN.markdown` file.
Welcome to AML contribution guide.
This page walk through the main library expectations.
When contributing to AML, contributors should write
features aligned with AML goal and AML spirit.
Additionnally, several good practices are requested, some beeing inforced
through the continuous integration (CI) pipeline.
For readability and maintainability, coding conventions need to be respected,
code needs to be tested, documented and integrated with the main AML branch
and its build tool-chain.
## Contribution Workflow
## General Guidelines
At its core, AML is able to provide access to established, well known
abstractions inside a common toolbox, with the more flexibility possible.
Abstractions are made available through __building blocks__, each representing
a single abstraction. Each building block is separated into a __generic__ API
for operations that the abstraction provides, and __implementations__ of those
operations for a specific implementation of the abstraction.
__generic__ API, i.e the first layer exposed to users
through a generic interface declares a table of operations common to the
abstraction, the main abstraction structure containing a such a table
and an opaque handle to store implementation specific data.
```
struct aml_building_block_ops{
aml_building_block_op0();
...
};
struct aml_building_block_data;
struct aml_building_block{
struct aml_building_block_ops *ops;
struct aml_building_block_data *data;
};
```
A set of generic functions taking the abstraction as first argument
is supposed to be exposed while it will de-reference underlying
implementation function table to implement expected behavior.
Such generic functions are supposed to return an AML error code as defined
in [error header](@ref error.h).
```
int aml_building_block_feature(struct aml_building_block *, ...);
```
Building blocks should interact with each other through the generic API, but
specific implementations are allowed to break this rule for
performance-oriented reasons. A generic version should still be available.
It is expected that these will be transparent and composable,
because low-level optimizations require control.
Therefore, most implementation details (structures, functions) will be exposed
to the user in a dedicated header, and use of opaque structures, static functions
and variables there should be avoided.
Moreover, customization of a building block should happen at creation time: the goal
is to make it easy for users to evaluate different configurations options with
minimal impact on the inner code of the application.
Therefore, building blocks specialized implementation, must declare a dynamic allocator, free,
initialization and finish functions, and stack declaration.
```
int aml_building_block_create (struct aml_building_block **, ...);
void aml_building_block_destroy(struct aml_building_block *);
int aml_building_block_init (struct aml_building_block *, ...);
void aml_building_block_fini (struct aml_building_block *);
#define AML_BUILDING_BLOCK_DECL(name) \
struct aml_building_block_data __ ##name## _inner_data; \
struct aml_building_block name = { \
&aml_building_block_ops, \
(struct aml_building_block_data *)&__ ## name ## _inner_data, \
}
```
Usually a building blocks implementation would declare an exportable (`extern`)
`struct aml_building_block_ops`, and `struct aml_building_block`, to provide
users with a default static implementation of the block.
Finally, building blocks headers, sources and tests go to a dedicated folder
located respectively in `include`, `src` and `tests`.
## Coding Convention
AML code respects several coding conventions. Some of them are enforced through
the CI pipeline.
Prior to start coding AML, you might want to set your editor to minimize
the process of code formatting.
Most of the formatting can also be scripted with `indent` program.
Code formatting compliance with AML requirements can be checked by running
[linux kernel linter](https://github.com/torvalds/linux/blob/master/scripts/checkpatch.pl)
at the root of AML project.
The list of coding convention is the following:
* use tabulation of 8 characters,
* remove trailing white spaces,
* lines may not exceed 80 characters,
* pointers star have to stick to the variable name: `void *var;`,
* function return type is on the same line as function name,
* curly braces are on the same line as the statement or function,
* A new line must appear after curly braces.
* macros are UPPERCASE_WITH_UNDERSCORES, function lowercase_with_underscores
* All functions/macros should start with aml/AML followed by the name of the building block involved.
## Versioning and Submission Workflow
AML code and CI runners is hosted on Argonne National Laboratory
[gitlab](https://xgitlab.cels.anl.gov/argo/aml) infrastructure under git version control.
New user may create new branches upon request to host and test their code.
New branched must be created from the master branch and merging is done
via git or gitlab pull request interface.
We basically follow the [Gitlab Flow](https://docs.gitlab.com/ee/workflow/gitlab_flow.html)
model:
- `master` is the primary branch for up-to-date development
- any new feature/development should go into a `feature-branch` based of
- any new feature/development should go into a `feature-branch` based on
`master` and merge into `master` after.
- we follow a `semi-linear history with merge commits` strategy: each merge
request should include clean commits, but only the HEAD and merge commit
......@@ -21,7 +126,8 @@ model:
change should appear in its own commit.
- to help with in-development work, CI will not activate on branches with
names started with wip/
## Commit Messages Styleguide
### Commit Messages Styleguide
- use present tense, imperative mood
- reference issues and merge requests
......@@ -59,7 +165,7 @@ commit.template my-config-template.txt` to use it automatically.
# --------------------
```
## Signoff on Contributions:
### Signoff on Contributions:
The project uses the [Developer Certificate of
Origin](https://developercertificate.org/) for copyright and license
......@@ -69,3 +175,98 @@ the `LICENSE` file) and that you agree to the DCO.
To signoff commits, use: `git commit --signoff`.
To signoff a branch after the fact: `git rebase --signoff`
## Testing
AML project includes a set of unit and integration tests.
Each AML building block has its own set of tests in a separate directory.
Tests are launched together with automake test suite with the `make check` command.
Building blocks should be tested as thoroughly as possible and whenever
relevant, integration tests with other components should be available.
Tests compilation and launch are implemented together in `tests/Makefile.am`.
Whenever a branch is pushed to AML repository, it automatically triggers
the CI pipeline, unless branch name starts with `wip/`.
The CI pipeline must be successful for a merge request to be accepted.
## Documentation
For obvious reasons, the code and building block must be documented.
The whole documentatation is based doxygen tool.
All the public code (data structures, functions, macro, variables) must be
documented as precisely as possible. Additionnaly, new buiding blocks implementation
must create a group (with `\@defgroup`) and provide a detailed description of
the features it provides. Syntax for code documentation can be found
[here](http://www.doxygen.nl/manual/docblocks.html),
and most useful elements can be found in the example below.
```
/**
* \@defgroup aml_buidling_block_implementation "AML buidling block implemation"
*
* \@brief This is a short description of this group.
*
* The documentation encapsulated below
* including this long description of the building_block
* implementation will be create a section dedicated to this buidling block
* implementation. It will be possible to link to it in the main header
* documentation with "@ref" or "@see".
* \@{
**/
/**
* This is a default operation table implementing
* table defined in main header.
**/
extern struct aml_building_block_ops aml_building_block_ops;
/**
* This is the bb inner data declaration, to
* stuff with needed attributes.
**/
struct aml_building_block_data {};
/**
* This is the bb declaration
**/
struct aml_building_block{
struct aml_building_block_ops *ops;
struct aml_building_block_data *data;
};
/** This is the static buidling block declaration macro **/
#define AML_BUILDING_BLOCK_DECL(name) \
struct aml_building_block_data __ ##name## _inner_data; \
struct aml_building_block name = { \
&aml_building_block_ops, \
(struct aml_building_block_data *)&__ ## name ## _inner_data, \
}
/** Static declaration of the building block size **/
#define AML_BUILDING_BLOCK_ALLOCSIZE (sizeof(struct aml_building_block_data) + \
sizeof(struct aml_building_block))
/**
* Allocates and initializes a new building_block.
*
* \@param bb[out]: A pointer to store the new building block.
* \@param other[in]: Other parameters used as input.
* \@return AML_SUCCESS if successful; an error code otherwise.
**/
int aml_building_block_create(struct aml_building_block **bb, ...);
...
/**
* This is the end of the inclusion into building_block documentation group.
* \@}
**/
```
If the contribution is an implementation of a building block and header has
been placed in `include/aml/building_block` then its documentation will automatically added to
the project documentation. However, it is necessary to point to this particular group
with `@ref` or `@see` command in the [main aml header](\ref aml.h).
If the contribution creates new include directories, the latter have to be included
in `doc/aml.doxy` after the field `INPUT`.
......@@ -5,7 +5,12 @@ if ADD_BENCHMARKS
SUBDIRS += benchmarks
endif
if AML_DOXYGEN
SUBDIRS += doc
endif
pkgconfigdir = $(libdir)/pkgconfig
pkgconfig_DATA = aml.pc
EXTRA_DIST = autogen.sh aml.pc README.markdown
......@@ -7,6 +7,30 @@ memory management policies for allocation and placement of data across devices.
This library is still in the prototyping phase. APIs might break often.
## Documentation
The latest documentation can be found
[online](https://argo-aml.readthedocs.io/en/latest/),
can be installed when installing a release or generated
from git clone before installing with
```
make -C doc doxygen
```
## Requirements:
* autoconf
* automake
* libnuma
## Installation
```
sh autogen.sh
./configure
make -j install
```
# General Architecture
The architecture of the library relies on two principles: type-specific
......
......@@ -69,6 +69,12 @@ AC_CHECK_HEADERS([numa.h],,[AC_MSG_ERROR([AML requires libnuma headers.])])
AC_CHECK_HEADERS([numaif.h],,[AC_MSG_ERROR([AML requires libnuma headers.])])
AC_CHECK_LIB(numa, mbind,,[AC_MSG_ERROR([AML requires libnuma.])])
# Doxygen
# See m4/ax_doxygen.m4
AC_CHECK_PROG(doxygen_happy, [doxygen], [yes], [no])
AM_CONDITIONAL([AML_DOXYGEN],[ test "x$doxygen_happy" = "xyes" ])
AC_CONFIG_HEADERS([include/config.h])
AC_CONFIG_FILES([Makefile
......@@ -80,6 +86,7 @@ AC_CONFIG_FILES([Makefile
src/utils/Makefile
include/Makefile
tests/Makefile
doc/Makefile
benchmarks/Makefile
aml.pc
include/aml/utils/version.h])
......
DATA_INSTALL_DIR=$(datadir)/aml
DOXYGEN_BUILD_DIR=./build-doxygen
SPHINX_BUILD_DIR=./build-sphinx
doxygen:
doxygen aml.doxy
create-doc-dir:
mkdir -p $(DATA_INSTALL_DIR)
install-doc-html:
if [ -d $(DOXYGEN_BUILD_DIR)/html ]; then \
cp -r $(DOXYGEN_BUILD_DIR)/html $(DATA_INSTALL_DIR); \
fi
install-data-local: create-doc-dir install-doc-html
uninstall-local:
rm -rf $(DATA_INSTALL_DIR)
clean-local:
rm -rf $(DOXYGEN_BUILD_DIR) $(SPHINX_BUILD_DIR)
dist-hook: doxygen
cp -r $(DOXYGEN_BUILD_DIR) $(distdir)
This source diff could not be displayed because it is too large. You can view the blob instead.
################################################################################
# Copyright 2019 UChicago Argonne, LLC.
# (c.f. AUTHORS, LICENSE)
#
# This file is part of the AML project.
# For more info, see https://xgitlab.cels.anl.gov/argo/aml
#
# SPDX-License-Identifier: BSD-3-Clause
################################################################################
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# http://www.sphinx-doc.org/en/master/config
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
import subprocess
# Generate documentation
subprocess.call('doxygen aml.doxy', shell=True)
# -- General configuration ---------------------------------------------------
# The language for content autogenerated by Sphinx.
language = 'C'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = []
# Add any paths that contain templates here, relative to this directory.
templates_path = []
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []
# -- Project information -----------------------------------------------------
project = 'AML'
copyright = 'Copyright 2019 UChicago Argonne, LLC'
author = 'Swann Perarnau, Kamil Iskra, Brian Suchy, Valentin Reis, Nicolas Denoyelle'
release = 'X.X.X'
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = []
#Overwrite sphinx documentation with doxygen documentation.
html_extra_path = ['build-doxygen/html']
This is an empty file required by shpinx when
compiling documentation on readthedocs.org.
This is an empty landing page overwritten with doxygen documentation.
Welcome to AML Documentation Page {#index}
============
AML is a memory management library designed for highly optimized
codes to delegate complex memory operations.
## Overview
AML goals are the following:
1. to provide a convenient interface for addressing all heterogeneous
addressable memories and moving data around.
2. to provide an interface for expressing and translating data
structures between several representations and physical memories.
Toward achieving these goals, AML implements the following abstractions:
\image html building-blocks-diagram.png width=400cm
* [Areas](@ref aml_area), a set of addressable physical memories,
* [Tilings](@ref aml_tiling), a description of tiling data structures,
* [DMA](@ref aml_dma), an engine to asynchronously move data structures between areas,
* [Scratchpad](@ref aml_scratch), a stage-in, stage-out abstraction for prefetching.
Each of these abstractions have several implementations. For instance,
areas may refer to usual DDR of a subset of them, GPU memory or non-volatile memory.
Tilings are implemented to reflect either 1D or 2D structures etc.
## Problems Solved with AML
AML library is designed for highly optimized codes to delegate complex memory
operations. AML is currently used in a
[proof of concept](https://xgitlab.cels.anl.gov/argo/nauts/tree/master/papers/mchpc18)
of efficient matrix multiplication on architectures with software managed memory side
cache. It is also currently being extended toward specialized memory allocations
for latency sensitive applications.
## Quick Start Guide
### Download
* [Github version](https://xgitlab.cels.anl.gov/argo/aml):
```
git clone git@xgitlab.cels.anl.gov:argo/aml.git
```
* Release versions:
TODO after release.
### Requirements:
* autoconf
* automake
* libnuma
### Installation
```
sh autogen.sh
./configure
make -j install
```
### Workflow
* Include aml header:
```
#include <aml.h>
...
int main(int argc, char **argv){
```
* Check AML version:
```
if(aml_version_major != AML_VERSION_MAJOR){
printf("AML ABI mismatch!");
return 1;
}
```
* Initialize and Cleanup AML:
```
if(aml_init(&argc, &argv) != 0){
printf("AML library init failure!");
return 1;
}
...
aml_finalize();
```
* Link your program with `-laml`
See above building blocks specific pages for further examples and information on
library features.
## Support
AML interface with users is mainly done through
[gitlab issue interface](https://xgitlab.cels.anl.gov/argo/aml/issues) and
direct contact with developers/maintainers:
* Swann Perarnau (swann@anl.gov)
* Nicolas Denoyelle (ndenoyelle@anl.gov)
AML is looking forward to new users. If you think your problem can be solved with AML
eventually with some additional non-existing features which are relevant to the library,
please let us know. Tell us about your topic of interests and your scientific/technical
problems. We are also pleased to improve the software, thus any suggestion, bug
declaration, contribution is welcomed.
## Contributing
AML is looking forward to new collaborations and contributions.
If you wish to contribute to AML project rendez-vous [here](@ref contributing_page)
......@@ -8,6 +8,13 @@
* SPDX-License-Identifier: BSD-3-Clause
*******************************************************************************/
/**
* \file aml.h
*
* \brief Main AML header file, contains all high level
* abstractions declarations.
**/
#ifndef AML_H
#define AML_H 1
......@@ -23,469 +30,825 @@
#include <sys/mman.h>
#include <unistd.h>
/* Used by mbind */
#ifndef PAGE_SIZE
#define PAGE_SIZE 4096
#endif
#include "aml/utils/bitmap.h"
#include "aml/utils/error.h"
#include "aml/utils/vector.h"
#include "aml/utils/version.h"
/*******************************************************************************
* General functions:
* Initialize internal structures, cleanup everything at the end.
******************************************************************************/
////////////////////////////////////////////////////////////////////////////////
/*
/**
* @}
* @defgroup aml "AML Library functions"
* @brief Initialize/Finalize library.
*
* General functions of aml library.
* Initialization of internal structures, cleanup of everything at the end.
*
* @see aml_error
* @see aml_version
* @{
**/
////////////////////////////////////////////////////////////////////////////////
/**
* Initializes the library.
* "argc": pointer to the main()'s argc argument; contents can get modified.
* "argv": pointer to the main()'s argv argument; contents can get modified.
* Returns 0 if successful; an error code otherwise.
*/
* @param argc: pointer to the main()'s argc argument; contents can get
* modified.
* @param argv: pointer to the main()'s argv argument; contents can get
* modified.
* @return 0 if successful; an error code otherwise.
**/
int aml_init(int *argc, char **argv[]);
/*
/**
* Terminates the library.
* Returns 0 if successful; an error code otherwise.
*/
* @return 0 if successful; an error code otherwise.
**/
int aml_finalize(void);
/*******************************************************************************
* Areas:
* embeds information about a byte-addressable physical memory location as well
* as binding policies over it.
******************************************************************************/
////////////////////////////////////////////////////////////////////////////////
/**
* An AML area is an implementation of memory operations for several type of
* devices through a consistent abstraction.
* This abstraction is meant to be implemented for several kind of devices,
* i.e the same function calls allocate different kinds of devices depending
* on the area implementation provided.
* List of aml area implementations each may provide additional functionalities:
* - <aml/area/linux.h>
* @}
* @defgroup aml_area "AML Area"
* @brief Area High-Level API
*
* AML areas represent places where data can belong.
* In shared memory systems, locality is a major concern for performance.
* Beeing able to query memory from specific places is of a major interest
* two achieve this goal. AML Areas provide mmap / munmap low level functions
* to query memory from specific places materialized as areas. Available area
* implementations dictate the way such places can be arranged and with which
* properties. It is important to notice that function provided through Area API
* are low-level functions and are not optimized for performance as allocator
* are.
*
* @image html area.png "Illustration of areas on a copmlex system." width=700cm
*
* @see aml_area_linux
*
* @{
**/
/* Opaque handle to areas data. Defined by implementations */
////////////////////////////////////////////////////////////////////////////////
/**
* aml_area_data is an opaque handle defined by each aml_area
* implementation. This not supposed to be used by end users.
**/
struct aml_area_data;
/** Implementation specific operations. **/
/**
* aml_area_ops is a structure containing implementations
* of an area operations.
* Aware users may create or modify implementation by assembling
* appropriate operations in such a structure.
**/
struct aml_area_ops {
/**
* Coarse grain allocator of virtual memory.
* Building block for coarse grain allocator of virtual memory.
*
* "area_data": Opaque handle to implementation specific data.
* "ptr": A virtual address to be used by underlying implementation.
* @param data: Opaque handle to implementation specific data.
* @param ptr: A virtual address to be used by underlying
* implementation.
* Can be NULL.
* "size": The minimum size of allocation.
* Is greater than 0. Must not fail unless not enough
* memory is available, or ptr argument does not point to a
* suitable address.
* In case of failure, aml_errno must be set to an appropriate
* value.
*
* Returns a pointer to allocated memory object.
* @param size: The minimum size of allocation.
* Is greater than 0. Must not fail unless not enough
* memory is available, or ptr argument does not point to a
* suitable address.
* In case of failure, aml_errno must be set to an appropriate
* value.
* @return a pointer to allocated memory object.
**/
void* (*mmap)(const struct aml_area_data *area_data,
void* (*mmap)(const struct aml_area_data *data,
void *ptr,
size_t size);
/**
* Unmapping of virtual memory mapped with map().
*
* "area_data": An opaque handle to implementation specific data.
* "ptr": Pointer to data mapped in physical memory. Cannot be NULL.
* "size": The size of data. Cannot be 0.
* Building block for unmapping of virtual memory mapped with mmap()
* of the same area.
*
* Returns AML_AREA_* error code.
* @param data: An opaque handle to implementation specific data.
* @param ptr: Pointer to data mapped in physical memory. Cannot be
* NULL.
* @param size: The size of data. Cannot be 0.
* @return: AML_AREA_* error code.
* @see mmap()
**/
int (*munmap)(const struct aml_area_data *area_data,
int (*munmap)(const struct aml_area_data *data,
void *ptr,
size_t size);
};
/**