GitLab maintenance scheduled for Tomorrow, 2019-03-26, from 12:00 to 13:00 CDT - Services will be unavailable during this time.

Commit bb75028f authored by Shane Snyder's avatar Shane Snyder

remove old modularization design docs

parent 6ccaea46
Rough design notes on modularizing Darshan
- Darshan is split into two parts (subdirs in the same repo):
- runtime: runtime instrumentation for MPI programs
- util: post-processing of logs
Runtime design
- current code has the following responsibilities:
- init:
- set up data structures
- during runtime:
- track file names and handles
- memory allocation
- intercepting function calls
- updating counters
- shutdown:
- identify shared files
- aggregation/reduction
- compression
- write log
- propose division of code in modular runtime library:
(these aren't literally separate libraries, they are probably all
- core lib:
- central component that modules register with, coordinates shutdown
- modules:
- posix, mpi-io, pnetcdf, hdf5, asg, etc.
- register with the core lib and track statistics for a single API
- common/utility lib:
- contains utility functions
- not mandatory for a module to use this, but may make things easier
- responsibilities of core library:
- track file names and map them to generic IDs
(keep full path names)
- tell modules how much memory they can consume
- kick off shutdown procedure
- perform generic (zlib) compression
- at shutdown time, the core library will:
- create output file
- write header and index information
- write out filename->ID mapping
- perform its own aggregation step to identify files shared across ranks
- core API (presented by core library, used by modules):
- register(const char* name, int* runtime_mem_limit, struct mod_fns *mfns)
- lets module register with the core library, provide its name and table
of function pointers, and get back a limit on how much RAM it can
- lookup_id(void* name, int len, int64* ID, int printable_flag);
- used by module to convert a file name to a generic ID. printable_flag
tells Darshan that the "name" is not a string (as in ASG use case)
- module API (will be function pointers in struct mod_fns above, this is the
API that each module must present to the core library)
- prep_for_shutdown()
- tells the module that it should stop instrumenting and perform any
module-specific aggregation or custom compression that it wants to do
before Darshan stores its results
- get_output(void **buffer, int size)
- called by core library to get a pointer to the data that should be
written into the log file. Darshan will zlib compress it and put it
in the right position in the output file.
- how will the asg module fit in?
- it doesn't have file names
- will pass in object IDs instead that will still get mapped to generic
Darshan IDs just like a file name would have
- set flag telling Darshan that the "name" won't be printable
- compiler script:
- how much do we want to modularize here?
- don't need to do this initially, but we could have the compiler script
call out to a predefined directory to look for scripts or files that let
each module describe the linker arguments to add
- avoid extremely large ld arguments
- utility library:
- this is the part run to process existing logs
- file format:
- header (endianness, version number, etc.)
- job information (cmd line, start time, end time, etc.)
- indices
- location/size of name->id mapping table
- location/size of each module's opaque data (with name)
- table of name->id mapping
- needs to handle variable length names (some of which won't be
- format it however makes sense for parsing
- compress this part since it will often contain mostly text
- opaque blobs containing data for each module
- modules will refer to files using ID from name->id table, won't
store full paths here
- each module can define its own parser, grapher, etc. as needed
- for convenience we may integrate posix and mpi-io support into the default
darshan tools
- development notes
- do development in git branch
- ignore compatibility (we'll work that out later)
- strip down to basic example
- just do one or two posix counters to start, but exercise all of the
API and code organization stuff
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment