darshan-modularization.txt 26.5 KB
Newer Older
1
:data-uri:
Shane Snyder's avatar
Shane Snyder committed
2

3 4
Darshan modularization branch development notes
===============================================
5 6 7

== Introduction

8 9
Darshan is a lightweight toolkit for characterizing the I/O performance of instrumented
HPC applications.
10

11 12 13 14 15 16 17
Starting with version 3.0.0, the Darshan runtime environment and log file format have
been redesigned such that new "instrumentation modules" can be added without breaking
existing tools. Developers are given a framework to implement arbitrary instrumentation
modules, which are responsible for gathering I/O data from a specific system component
(which could be from an I/O library, platform-specific data, etc.). Darshan can then
manage these modules at runtime and create a valid Darshan log regardless of how many
or what types of modules are used.
18

19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
== Checking out and building the modularization branch

Developers can clone the Darshan source repository using the following methods:

* "git clone git://git.mcs.anl.gov/radix/darshan" (read-only access)

* "git clone \git@git.mcs.anl.gov:radix/darshan" (authenticated access)

After cloning the Darshan source, it is necessary to checkout the modularization
development branch:

----
git clone git@git.mcs.anl.gov:radix/darshan
git checkout dev-modular
----

35 36
For details on configuring and building the Darshan runtime and utility repositories,
consult the documentation from previous versions
37 38 39 40 41
(http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-runtime.html[darshan-runtime] and
http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-util.html[darshan-util]) -- the
necessary steps for building these repositories should not have changed in the new version of
Darshan.

42
== Darshan dev-modular overview
43

44
The Darshan source tree is organized into two primary components:
45 46 47 48 49 50 51 52 53 54

* *darshan-runtime*: Darshan runtime environment necessary for instrumenting MPI
applications and generating I/O characterization logs.

* *darshan-util*: Darshan utilities for analyzing the contents of a given Darshan
I/O characterization log.

The following subsections provide an overview of each of these components with specific
attention to how new instrumentation modules may be integrated into Darshan.

55
=== Darshan-runtime
56

57
The primary responsibilities of the darshan-runtime component are:
58

59
* intercepting I/O functions of interest from a target application;
60

61
* extracting statistics, timing information, and other data characterizing the application's I/O workload;
62

63
* compressing I/O characterization data and corresponding metadata;
64

65
* logging the compressed I/O characterization to file for future evaluation
66

Shane Snyder's avatar
Shane Snyder committed
67 68 69 70 71 72 73 74 75 76
The first two responsibilities are the burden of module developers, while the last two are handled
automatically by Darshan.

In general, instrumentation modules are composed of:

* wrapper functions for intercepting I/O functions;

* internal functions for initializing and maintaining internal data structures and module-specific
  I/O characterization data;

77
* a set of functions for interfacing with the Darshan runtime environment
78

79 80 81 82 83 84
A block diagram illustrating the interaction of an example POSIX instrumentation module and the
Darshan runtime environment is given below in Figure 1.

.Darshan runtime environment
image::darshan-dev-modular-runtime.png[align="center"]

85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108
As shown in Figure 1, the Darshan runtime environment is just a library (libdarshan) which
intercepts and instruments functions of interest made by an application to existing system
libraries. Two primary components of this library are `darshan-core` and `darshan-common`.
`darshan-core` is the central component which manages the initialization/shutdown of Darshan,
coordinates with active instrumentation modules, and writes I/O characterization logs to disk,
among other things. `darshan-core` intercepts `MPI_Init()` to initialize key internal data
stuctures and intercepts `MPI_Finalize()` to initiate Darshan's shutdown process. `darshan-common`
simply provides module developers with functionality that is likely to be reused across modules
to minimize development and maintenance costs. Instrumentation modules must utilize `darshan-core`
to register themselves and corresponding I/O records with Darshan so they can be added to the
output I/O characterization. While not shown in Figure 1, numerous modules can be registered
with Darshan at any given time and Darshan is capable of correlating records between these
modules.

In the next three subsections, we describe instrumentation modules, the `darshan-core` component,
and the `darshan-common` component in more detail. In
link:darshan-modularization.html#_adding_new_instrumentation_modules[Section 4], we provide the
required steps for integrating new instrumentation modules into Darshan. This section also includes
details on an example module that can serve as a minimal starting point for new module implementations.
In link:darshan-modularization.html#_shared_record_reductions[Section 5], we provide details on
implementing a shared record reduction operation within an instrumentation module. This optional
mechanism allows modules to compress records which are shared across all processes of an application,
minimizing the size of the resulting I/O characterization log.

109 110 111 112
==== Instrumentation modules

The wrapper functions used to intercept I/O function calls of interest are central to the design of
any Darshan instrumentation module. These wrappers are used to extract pertinent I/O data from
Shane Snyder's avatar
Shane Snyder committed
113 114 115 116
the function call and persist this data in some state structure maintained by the module. Modules
must bootstrap themselves by initializing internal data structures within wrapper functions. The
wrappers are inserted at compile time for statically linked executables (e.g., using the linkers
`--wrap` mechanism) and at runtime for dynamically linked executables (using LD_PRELOAD).
117 118 119 120

*NOTE*: Modules should not perform any I/O or communication within wrapper functions. Darshan records
I/O data independently on each application process, then merges the data from all processes when the
job is shutting down. This defers expensive I/O and communication operations to the shutdown process,
Shane Snyder's avatar
Shane Snyder committed
121
minimizing Darshan's impact on application I/O performance.
122 123 124 125 126 127 128 129 130

When the instrumented application terminates and Darshan begins its shutdown procedure, it requires
a way to interface with any active modules that have data to contribute to the output I/O characterization.
Darshan requires that module developers implement the following functions to allow the Darshan runtime
environment to coordinate with modules while shutting down:

[source,c]
struct darshan_module_funcs
{
Shane Snyder's avatar
Shane Snyder committed
131
    void (*begin_shutdown)(void);
132 133 134 135 136 137
    void (*get_output_data)(
        void** buf,
        int* size
    );
    void (*shutdown)(void);
    /* OPTIONAL: shared record reductions ignored by setting setup_reduction to NULL */
Shane Snyder's avatar
Shane Snyder committed
138
    void (*setup_reduction)(
139 140 141 142 143 144
        darshan_record_id *shared_recs,
        int *shared_rec_count,
        void **send_buf,
        void **recv_buf,
        int *rec_size
    );
145
    /* OPTIONAL: shared record reductions ignored by setting record_reduction_op to NULL */
Shane Snyder's avatar
Shane Snyder committed
146
    void (*record_reduction_op)(
147 148 149 150 151 152 153
        void* a,
        void* b,
        int *len,
        MPI_Datatype *datatype
    );
};

Shane Snyder's avatar
Shane Snyder committed
154
`begin_shutdown()`
155 156

This function informs the module that Darshan is about to begin shutting down. It should disable
Shane Snyder's avatar
Shane Snyder committed
157 158 159 160
all wrappers to prevent the module from making future updates to internal data structures, primarily
to ensure data consistency and avoid other race conditions. This function also serves as a final
opportunity for a module to modify internal data structures prior to a possible reduction of shared
data.
161 162 163 164

`get_output_data()`

This function is responsible for passing back a single buffer storing all data this module is
Shane Snyder's avatar
Shane Snyder committed
165
contributing to the output I/O characterization.
166 167 168 169 170 171 172 173 174 175 176

* _buf_ is a pointer to the address of the buffer this module is contributing to the I/O
characterization. 

* _size_ is the size of this module's output buffer.

`shutdown()`

This function is a signal from Darshan that it is safe to shutdown. It should clean up and free
all internal data structures.

177 178 179 180 181
`setup_reduction()` and `record_reduction_op()` are optional function pointers which can be
implemented to utilize Darshan's shared I/O record reduction mechanism, described in detail in
link:darshan-modularization.html#_shared_record_reductions[Section 5]. Module developers can
bypass this mechanism by setting these function pointers to NULL. 

182 183 184 185
==== darshan-core

Within darshan-runtime, the darshan-core component manages the initialization and shutdown of the
Darshan environment, provides instrumentation module developers an interface for registering modules
Shane Snyder's avatar
Shane Snyder committed
186
with Darshan, and manages the compressing and the writing of the resultant I/O characterization.
187 188 189 190 191
As illustrated in Figure 1, the darshan-core runtime environment intercepts `MPI_Init` and
`MPI_Finalize` routines to initialze and shutdown the darshan runtime environment, respectively.

Each of the functions provided by darshan-core to interface with instrumentation modules are
described in detail below.
192 193 194 195 196

[source,c]
void darshan_core_register_module(
    darshan_module_id mod_id,
    struct darshan_module_funcs *funcs,
Shane Snyder's avatar
Shane Snyder committed
197 198
    int *mod_mem_limit,
    int *sys_mem_alignment);
199 200 201 202 203 204 205 206 207 208 209

The `darshan_core_register_module` function registers Darshan instrumentation modules with the
darshan-core runtime environment. This function needs to be called at least once for any module
that will contribute data to Darshan's final I/O characterization. 

* _mod_id_ is a unique identifier for the given module, which is defined in the Darshan log
format header file (darshan-log-format.h).

* _funcs_ is the structure of function pointers (as described above) that a module developer must
provide to interface with the darshan-core runtime. 

Shane Snyder's avatar
Shane Snyder committed
210
* _mod_mem_limit_ is a pointer to an integer which will store the amount of memory Darshan
211 212 213 214
allows this module to use at runtime. Currently, darshan-core will hardcode this value to 2 MiB,
but in the future this may be changed to optimize Darshan's memory footprint. Note that Darshan
does not allocate any memory for modules, it just informs a module how much memory it can use.

Shane Snyder's avatar
Shane Snyder committed
215 216 217 218
* _sys_mem_alignment_ is a pointer to an integer which will store the system memory alignment value
Darshan was configured with. This parameter may be set to `NULL` if a module is not concerned with the
memory alignment value.

219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236
[source,c]
void darshan_core_unregister_module(
    darshan_module_id mod_id);

The `darshan_core_unregister_module` function disassociates the given module from the
darshan-core runtime. Consequentially, Darshan does not interface with the given module at
shutdown time and will not log any I/O data from the module. This function should only be used
if a module registers itself with darshan-core but later decides it does not want to contribute
any I/O data.

* _mod_id_ is the unique identifer for the module being unregistered.

[source,c]
void darshan_core_register_record(
    void *name,
    int len,
    int printable_flag,
    darshan_module_id mod_id,
Shane Snyder's avatar
Shane Snyder committed
237 238
    darshan_record_id *rec_id,
    int *file_alignment);
239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254

The `darshan_core_register_record` function registers some data record with the darshan-core
runtime. This record could reference a POSIX file or perhaps an object identifier for an
object storage system, for instance.  A unique identifier for the given record name is
generated by Darshan, which should then be used by the module for referencing the corresponding
record.  This allows multiple modules to refer to a specific data record in a consistent manner
and also provides a mechanism for mapping these records back to important metadata stored by
darshan-core. It is safe (and likely necessary) to call this function many times for the same
record -- darshan-core will just set the corresponding record identifier if the record has
been previously registered.

* _name_ is just the name of the data record, which could be a file path, object ID, etc.

* _len_ is the size of the input record name. For string record names, this would just be the
string length, but for nonprintable record names (e.g., an integer object identifier), this
is the size of the record name type.
255

256 257 258 259 260 261 262
* _printable_flag_ indicates whether the input record name is a printable ASCII string.

* _mod_id_ is the identifier for the module attempting to register this record.

* _rec_id_ is a pointer to a variable which will store the unique record identifier generated
by Darshan.

Shane Snyder's avatar
Shane Snyder committed
263 264 265 266
* _file_alignment_ is a pointer to an integer which will store the the file alignment (block size)
of the underlying storage system. This parameter may be set to `NULL` if it is not applicable to a
given module.

267 268 269 270 271
[source,c]
void darshan_core_unregister_record(
    darshan_record_id rec_id,
    darshan_module_id mod_id);

Shane Snyder's avatar
Shane Snyder committed
272
The `darshan_core_unregister_record` function disassociates the given module identifier from the
273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300
given record identifier. If no other modules are associated with the given record identifier, then
Darshan removes all internal references to the record. This function should only be used if a
module registers a record with darshan-core, but later decides not to store the record internally.

* _rec_id_ is the record identifier we want to unregister.

* _mod_id_ is the module identifier that is unregistering _rec_id_.

[source,c]
double darshan_core_wtime(void);

The `darshan_core_wtime` function simply returns a floating point number of seconds since
Darshan was initialized. This functionality can be used to time the duration of application
I/O calls or to store timestamps of when functions of interest were called.

==== darshan-common

darshan-common is a utility component of darshan-runtime, providing module developers with
general functions that are likely to be reused across multiple modules. These functions are
distinct from darshan-core functions since they do not require access to internal Darshan
state.

[source,c]
char* darshan_clean_file_path(
    const char* path);

The `darshan_clean_file_path` function just cleans up the input path string, converting
relative paths to absolute paths and suppressing any potential noise within the string.
301
The address of the new string is returned and should be freed by the user.
302 303 304 305 306 307 308 309

* _path_ is the input path string to be cleaned up.

As more modules are contributed, it is likely that more functionality can be refactored out
of module implementations and maintained in darshan-common, facilitating code reuse and
simplifying maintenance.

=== Darshan-util
310

311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380
The darshan-util component is composed of a log parsing library (libdarshan-util) and a
corresponding set of utility programs that can parse and analyze Darshan I/O characterization
logs using this library. The log parsing library includes a generic interface (see
`darshan-logutils.h`) for retrieving specific portions of a given log file. Specifically,
this interface allows utilities to retrieve a log's header metadata, job details, record
identifier mapping, and any module-specific data contained within the log.

Module developers may wish to define additional interfaces for parsing module-specific data
that can then be integrated into the log parsing library. This extended functionality can be
implemented in terms of the generic functions offered by darshan-logutils and by module-specific
formatting information.

==== darshan-logutils

Here we define each function of the `darshan-logutils` interface, which can be used to implement
new log analysis utilities and to define module-specific functionality.

[source,c]
darshan_fd darshan_log_open(const char *name, const char* mode);

Opens Darshan log file stored at path `name`. `mode` can only be `r` for reading or `w` for
writing. Returns a Darshan file descriptor on success, or `NULL` on error.

[source,c]
int darshan_log_getheader(darshan_fd fd, struct darshan_header *header);

Fills in `header` structure from the log file referenced by descriptor `fd`. The `darshan_header`
structure is defined in `darshan-log-format.h`. Returns `0` on success, `-1` on failure.

[source,c]
int darshan_log_getjob(darshan_fd fd, struct darshan_job *job);

Fills in `job` structure from the log file referenced by descriptor `fd`. The `darshan_job`
structure is defined in `darshan-log-format.h`. Returns `0` on success, `-1` on failure.

[source,c]
int darshan_log_getexe(darshan_fd fd, char *buf);

Fills in `buf` with the corresponding executable string (exe name and command line arguments)
for the Darshan log referenced by `fd`. Returns `0` on success, `-1` on failure.

[source,c]
int darshan_log_getmounts(darshan_fd fd, char*** mnt_pts, char*** fs_types, int* count);

Returns mounted file system information for the Darshan log referenced by `fd`. `mnt_pnts` points
to an array of strings storing mount points, `fs_types` points to an array of strings storing file
system types (e.g., ext4, nfs, etc.), and `count` points to an integer storing the total number
of mounted file systems recorded by Darshan. Returns `0` on success, `-1` on failure.

[source,c]
int darshan_log_gethash(darshan_fd fd, struct darshan_record_ref **hash);

Builds hash table of Darshan record identifiers to full names for all records contained
in the Darshan log referenced by `fd`. `hash` is a pointer to the hash table (of type
struct darshan_record_ref *, which should be initialized to `NULL`). This hash table is
defined by the `uthash` hash table implementation and includes corresponding macros for
searching, iterating, and deleting records from the hash. For detailed documentation on using this
hash table, consult `uthash` documentation in `darshan-util/uthash-1.9.2/doc/txt/userguide.txt`.
The `darshan-posix-parser` utility (for parsing POSIX module information out of a Darshan log)
provides an example of how this hash table may be used. Returns `0` on success, `-1` on failure.

[source,c]
int darshan_log_get_moddat(darshan_fd fd, darshan_module_id mod_id, void *moddat_buf, int moddat_buf_sz);

Retrieves a chunk of (uncompressed) data for the module identified by `mod_id` from the Darshan log
referenced by `fd`. `moddat_buf_sz` specifies the number of uncompressed bytes to read from the file
and store in `moddat_buf`. This function may be repeatedly called to retrieve sequential chunks of data
from a given Darshan file descriptor. This function returns `1` if `moddat_buf_sz` bytes were read
successfully, `0` if no more data is available for this module, and `-1` otherwise.

381 382 383 384 385 386
*NOTE*: Darshan use a reader makes right conversion strategy to rectify endianness issues
between the machine a log was generated on and a machine analyzing the log. Accordingly,
module-specific log utility functions will need to check the `swap_flag` variable of the Darshan
file descriptor to determine if byte swapping is necessary. 32-bit and 64-bit byte swapping
macros (DARSHAN_BSWAP32/DARSHAN_BSWAP64) are provided in `darshan-logutils.h`.

387 388 389 390
[source,c]
void darshan_log_close(darshan_fd fd);

Close Darshan file descriptor `fd`. Returns `0` on success, `-1` on failure.
391 392 393

== Adding new instrumentation modules

Shane Snyder's avatar
Shane Snyder committed
394 395 396 397 398 399 400
In this section we outline each step necessary to adding a module to Darshan. To assist module
developers, we have provided the example "NULL" module (`darshan-runtime/lib/darshan-null.c`)
as part of the darshan-runtime source. This example can be used as a minimal stubbed out module
implementation. It is also heavily annotated to document more specific functionality provided
by Darshan to module developers. For a full-fledged implementation of a module, developers
can examine the POSIX module (`darshan-runtime/lib/darshan-posix.c`), which wraps and instruments
a number of POSIX I/O functions.
401 402 403 404 405 406 407 408 409

=== Log format headers

The following modifications to Darshan log format headers are required for defining
the module's record structure:

* Add module identifier to darshan_module_id enum and add module string name to the
darshan_module_name array in `darshan-log-format.h`.

Shane Snyder's avatar
Shane Snyder committed
410 411 412 413 414 415
* Add a top-level header that defines an I/O data record structure for the module. Consider
the "NULL" module and POSIX module log format headers for examples (`darshan-null-log-format.h`
and `darshan-posix-log-format.h`, respectively).

These log format headers are defined at the top level of the Darshan source tree, since both the
darshan-runtime and darshan-util repositories depend on them.
416 417 418 419 420 421 422 423

=== Darshan-runtime

==== Build modifications

The following modifications to the darshan-runtime build system are necessary to integrate
new instrumentation modules:

Shane Snyder's avatar
Shane Snyder committed
424 425 426 427 428
* Necessary linker flags for wrapping this module's functions need to be added to a
module-specific file which is used when linking applications with Darshan. For an example,
consider `darshan-runtime/darshan-posix-ld-opts`, the required linker options for the POSIX
module. The base linker options file for Darshan (`darshan-runtime/darshan-base-ld-opts.in`)
must also be updated to point to the new module-specific linker options file.
429 430

* Targets must be added to `Makefile.in` to build static and shared objects for the module's
Shane Snyder's avatar
Shane Snyder committed
431 432 433 434 435
source files, which will be stored in the `darshan-runtime/lib/` directory. The prerequisites
to building static and dynamic versions of `libdarshan` must be updated to include these objects,
as well.
    - If the module defines a linker options file, a target must also be added to install this
      file with libdarshan.
436 437 438

==== Instrumentation module implementation

Shane Snyder's avatar
Shane Snyder committed
439
In addtion to the development notes from above and the exemplar "NULL" and POSIX modules, we
440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461
provide the following notes to assist module developers:

* Modules only need to include the `darshan.h` header to interface with darshan-core.

* Lacking a way to bootstrap themselves, modules will have to include some logic in their
wrappers to initialize necessary module state if initialization has not already occurred.
    - Part of this initialization process should be registering the module with darshan-core,
    since this informs the module how much memory it may allocate.

* The file record identifier given when registering a record with darshan-core can be used
to store the record structure in a hash table or some other structure.
    - The `darshan_core_register_record` function is really more like a lookup function. It
    may be called multiple times for the same record -- if the record already exists, the function
    simply returns its record ID.
    - It may be necessary to maintain a separate hash table for other handles which the module
    may use to refer to a given record. For instance, the POSIX module may need to look up a
    file record based on a given file descriptor, rather than a path name.

=== Darshan-util

==== Build modifications

462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477
The following modifications to the darshan-util build system are necessary to integrate new
instrumentation modules:

* Update `Makefile.in` with new targets necessary for building new utilities and module-specific
log parsing source.
    - Make sure to update `all`, `clean`, and `install` targets to reference updates.
    - If adding new log parsing functionality, make sure to add it as a prerequisite source for 
    building libdarshan-util.

==== Module-specific logutils and utilities

For a straightforward reference implementation of module-specific log parsing functionality for
the POSIX module, consider files `darshan-posix-logutils.c` and `darshan-posix-logutils.h`. 

Also, the `darshan-posix-parser` source provides a simple example of a utility which can leverage
libdarshan-util for analyzing the contents of a given Darshan I/O characterization log.
478

Shane Snyder's avatar
Shane Snyder committed
479 480 481 482 483 484
== Shared record reductions

Since Darshan perfers to aggregate data records which are shared across all processes into a single
data record, module developers should consider implementing this functionality eventually, though it
is not strictly required. 

485 486
As mentioned previously, module developers must provide implementations for the `setup_reduction()`
and `record_reduction_op()` functions in the darshan_module_funcs structure to leverage Darshan's
Shane Snyder's avatar
Shane Snyder committed
487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542
shared record reduction mechanism. These functions are described in detail as follows:

[source,c]
void (*setup_reduction)(
    darshan_record_id *shared_recs,
    int *shared_rec_count,
    void **send_buf,
    void **recv_buf,
    int *rec_size
);

This function is used to prepare a module for performing a reduction operation. In general, this
just involves providing the input buffers to the reduction, and (on rank 0 only) providing output
buffer space to store the result of the reduction.

* _shared_recs_ is a set of Darshan record identifiers which are associated with this module.
These are the records which need to be reduced into single shared data records.

* _shared_rec_count_ is a pointer to an integer storing the number of shared records will
be reduced by this module. When the function is called this variable points to the number
of shared records detected by Darshan, but the module can decide not to reduce any number
of these records. Upon completion of the function, this variable should point to the number
of shared records to perform reductions on (i.e., the size of the input and output buffers).

* _send_buf_ is a pointer to the address of the send buffer used for performing the reduction
operation. Upon completion, this variable should point to a buffer containing *_shared_rec_count_
records that will be reduced.

* _recv_buf_ is a pointer to the address of the receive bufffer used for performing the reduction
operation. Upon completion, this variable should point to a buffer containing *_shared_rec_count_
records that will be reduced. This variable is only valid on the root process (rank 0). This
buffer address needs to be stored with module state, as it will be needed when retrieiving
the final output buffers from this module.

* _rec_size_ is just the size of the record structure being reduced for this module.

[source,c]
void (*record_reduction_op)(
    void* a,
    void* b,
    int *len,
    MPI_Datatype *datatype
);

This is the function which performs the actual shared record reduction operation. The prototype
of this function matches that of the user function provided to the MPI_Op_create function. Refer
to the http://www.mpich.org/static/docs/v3.1/www3/MPI_Op_create.html[documentation] for further
details.

Note that a module will likely need to clean up it's internal state after a reduction to get
all data records into a contiguous buffer, as Darshan requires. This can be done within the
`get_output_buffer()` function. 

Module developers can examine the POSIX module for a comprehensive (and monolithic) implementation
of the shared record reduction functionality.

543 544 545 546 547 548 549
== Other resources

* http://www.mcs.anl.gov/research/projects/darshan/[Darshan website]
* http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-runtime.html[darshan-runtime documentation]
* http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-util.html[darshan-util documentation]
* https://lists.mcs.anl.gov/mailman/listinfo/darshan-users[Darshan-users mailing list]
* https://trac.mcs.anl.gov/projects/darshan/report[Darshan trac page]