Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
Shane Snyder
darshan
Commits
bb75028f
Commit
bb75028f
authored
Jun 20, 2016
by
Shane Snyder
Browse files
remove old modularization design docs
parent
6ccaea46
Changes
2
Hide whitespace changes
Inline
Side-by-side
doc/darshan-modularization-design-notes.txt
deleted
100644 → 0
View file @
6ccaea46
Rough design notes on modularizing Darshan
2014-09-24
------------------------
- Darshan is split into two parts (subdirs in the same repo):
- runtime: runtime instrumentation for MPI programs
- util: post-processing of logs
Runtime design
----------------
- current code has the following responsibilities:
- init:
- set up data structures
- during runtime:
- track file names and handles
- memory allocation
- intercepting function calls
- updating counters
- shutdown:
- identify shared files
- aggregation/reduction
- compression
- write log
- propose division of code in modular runtime library:
(these aren't literally separate libraries, they are probably all
combined):
- core lib:
- central component that modules register with, coordinates shutdown
- modules:
- posix, mpi-io, pnetcdf, hdf5, asg, etc.
- register with the core lib and track statistics for a single API
- common/utility lib:
- contains utility functions
- not mandatory for a module to use this, but may make things easier
- responsibilities of core library:
- track file names and map them to generic IDs
(keep full path names)
- tell modules how much memory they can consume
- kick off shutdown procedure
- perform generic (zlib) compression
- at shutdown time, the core library will:
- create output file
- write header and index information
- write out filename->ID mapping
- perform its own aggregation step to identify files shared across ranks
API:
- core API (presented by core library, used by modules):
- register(const char* name, int* runtime_mem_limit, struct mod_fns *mfns)
- lets module register with the core library, provide its name and table
of function pointers, and get back a limit on how much RAM it can
consume
- lookup_id(void* name, int len, int64* ID, int printable_flag);
- used by module to convert a file name to a generic ID. printable_flag
tells Darshan that the "name" is not a string (as in ASG use case)
- module API (will be function pointers in struct mod_fns above, this is the
API that each module must present to the core library)
- prep_for_shutdown()
- tells the module that it should stop instrumenting and perform any
module-specific aggregation or custom compression that it wants to do
before Darshan stores its results
- get_output(void **buffer, int size)
- called by core library to get a pointer to the data that should be
written into the log file. Darshan will zlib compress it and put it
in the right position in the output file.
- how will the asg module fit in?
- it doesn't have file names
- will pass in object IDs instead that will still get mapped to generic
Darshan IDs just like a file name would have
- set flag telling Darshan that the "name" won't be printable
- compiler script:
- how much do we want to modularize here?
- don't need to do this initially, but we could have the compiler script
call out to a predefined directory to look for scripts or files that let
each module describe the linker arguments to add
- avoid extremely large ld arguments
- utility library:
- this is the part run to process existing logs
- file format:
- header (endianness, version number, etc.)
- job information (cmd line, start time, end time, etc.)
- indices
- location/size of name->id mapping table
- location/size of each module's opaque data (with name)
- table of name->id mapping
- needs to handle variable length names (some of which won't be
printable)
- format it however makes sense for parsing
- compress this part since it will often contain mostly text
- opaque blobs containing data for each module
- modules will refer to files using ID from name->id table, won't
store full paths here
- each module can define its own parser, grapher, etc. as needed
- for convenience we may integrate posix and mpi-io support into the default
darshan tools
- development notes
- do development in git branch
- ignore compatibility (we'll work that out later)
- strip down to basic example
- just do one or two posix counters to start, but exercise all of the
API and code organization stuff
doc/darshan-modularization-whiteboard.pdf
deleted
100644 → 0
View file @
6ccaea46
File deleted
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment