Commit 2d3f980d authored by Shane Snyder's avatar Shane Snyder
Browse files

update darshan-runtime doc to reflect non-mpi work

parent dee86d0c
...@@ -13,12 +13,19 @@ application that has been instrumented with Darshan will produce a single ...@@ -13,12 +13,19 @@ application that has been instrumented with Darshan will produce a single
log file each time it is executed. This log summarizes the I/O access patterns log file each time it is executed. This log summarizes the I/O access patterns
used by the application. used by the application.
The darshan-runtime instrumentation only instruments MPI applications (the The darshan-runtime instrumentation has traditionally only supported MPI
application must at least call `MPI_Init()` and `MPI_Finalize()`). However, applications (specifically, those that call `MPI_Init()` and `MPI_Finalize()`),
it captures both MPI-IO and POSIX file access. It also captures limited but, as of version 3.2.0, Darshan also supports instrumentation of non-MPI
information about HDF5 and PnetCDF access. Darshan also exposes an API that applications. Regardless of whether MPI is used, Darshan provides detailed
can be used to develop and add new instrumentation modules (for other I/O library statistics about POSIX level file accesses made by the application.
interfaces or to gather system-specific data, for instance), as detailed in In the case of MPI applications, Darshan additionally captures detals on MPI-IO
level access, as well as limited information about HDF5 and PnetCDF access.
Note that instrumentation of non-MPI applications is currently only supported
in Darshan's shared library, which applications must `LD_PRELOAD`.
Starting in version 3.0.0, Darshan also exposes an API that can be used to develop
and add new instrumentation modules (for other I/O library interfaces or to gather
system-specific data, for instance), as detailed in
http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-modularization.html[this document]. http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-modularization.html[this document].
Newly contributed modules include a module for gathering system-specific parameters Newly contributed modules include a module for gathering system-specific parameters
for jobs running on BG/Q systems, a module for gathering Lustre striping data for for jobs running on BG/Q systems, a module for gathering Lustre striping data for
...@@ -42,12 +49,12 @@ http://www.mcs.anl.gov/darshan[Darshan web site]. ...@@ -42,12 +49,12 @@ http://www.mcs.anl.gov/darshan[Darshan web site].
== Requirements == Requirements
* MPI C compiler * C compiler (preferrably GCC-compatible)
* zlib development headers and library * zlib development headers and library
== Compilation == Compilation
.Configure and build example .Configure and build example (with MPI support)
---- ----
tar -xvzf darshan-<version-number>.tar.gz tar -xvzf darshan-<version-number>.tar.gz
cd darshan-<version-number>/darshan-runtime cd darshan-<version-number>/darshan-runtime
...@@ -56,6 +63,15 @@ make ...@@ -56,6 +63,15 @@ make
make install make install
---- ----
.Configure and build example (without MPI support)
----
tar -xvzf darshan-<version-number>.tar.gz
cd darshan-<version-number>/darshan-runtime
./configure --with-mem-align=8 --with-log-path=/darshan-logs --with-jobid-env=PBS_JOBID --without-mpi CC=gcc
make
make install
----
.Explanation of configure arguments: .Explanation of configure arguments:
* `--with-mem-align=` (mandatory): This value is system-dependent and will be * `--with-mem-align=` (mandatory): This value is system-dependent and will be
used by Darshan to determine if the buffer for a read or write operation is used by Darshan to determine if the buffer for a read or write operation is
...@@ -77,7 +93,9 @@ file. See `./configure --help` for details. ...@@ -77,7 +93,9 @@ file. See `./configure --help` for details.
active Darshan instrumentation modules can collectively consume. active Darshan instrumentation modules can collectively consume.
* `--with-zlib=`: specifies an alternate location for the zlib development * `--with-zlib=`: specifies an alternate location for the zlib development
header and library. header and library.
* `CC=`: specifies the MPI C compiler to use for compilation. * `CC=`: specifies the C compiler to use for compilation.
* `--without-mpi`: disables MPI support when building Darshan - MPI support is
assumed if not specified.
* `--enable-mmap-logs`: enables the use of Darshan's mmap log file mechanism. * `--enable-mmap-logs`: enables the use of Darshan's mmap log file mechanism.
* `--disable-cuserid`: disables use of cuserid() at runtime. * `--disable-cuserid`: disables use of cuserid() at runtime.
* `--disable-ld-preload`: disables building of the Darshan LD_PRELOAD library * `--disable-ld-preload`: disables building of the Darshan LD_PRELOAD library
...@@ -147,22 +165,22 @@ administrators group ...@@ -147,22 +165,22 @@ administrators group
=== Instrumentation method === Instrumentation method
The instrumentation method to use depends on whether the executables The instrumentation method to use depends on whether the executables
produced by your MPI compiler are statically or dynamically linked. If you produced by your compiler are statically or dynamically linked. If you
are unsure, you can check by running `ldd <executable_name>` on an example are unsure, you can check by running `ldd <executable_name>` on an example
executable. Dynamically-linked executables will produce a list of shared executable. Dynamically-linked executables will produce a list of shared
libraries when this command is executed. libraries when this command is executed.
Most MPI compilers allow you to toggle dynamic or static linking via options Some compilers allow you to toggle dynamic or static linking via options
such as `-dynamic` or `-static`. Please check your MPI compiler man page such as `-dynamic` or `-static`. Please check your compiler man page
for details if you intend to force one mode or the other. for details if you intend to force one mode or the other.
== Instrumenting statically-linked applications == Instrumenting statically-linked MPI applications
Statically linked executables must be instrumented at compile time. Statically linked executables must be instrumented at compile time.
The simplest methods to do this are to either generate a customized The simplest methods to do this are to either generate a customized
MPI compiler script (e.g. `mpicc`) that includes the link options and MPI compiler script (e.g. `mpicc`) that includes the link options and
libraries needed by Darshan, or to use existing profiling configuration libraries needed by Darshan, or to use existing profiling configuration
hooks for existing MPI compiler scripts. Once this is done, Darshan hooks for MPI compiler scripts. Once this is done, Darshan
instrumentation is transparent; you simply compile applications using instrumentation is transparent; you simply compile applications using
the darshan-enabled MPI compiler scripts. the darshan-enabled MPI compiler scripts.
...@@ -224,9 +242,9 @@ add the necessary link options and libraries. Please see the ...@@ -224,9 +242,9 @@ add the necessary link options and libraries. Please see the
`darshan-gen-*` scripts for examples or contact the Darshan users mailing `darshan-gen-*` scripts for examples or contact the Darshan users mailing
list for help. list for help.
== Instrumenting dynamically-linked applications == Instrumenting dynamically-linked MPI applications
For dynamically-linked executables, darshan relies on the `LD_PRELOAD` For dynamically-linked executables, Darshan relies on the `LD_PRELOAD`
environment variable to insert instrumentation at run time. The executables environment variable to insert instrumentation at run time. The executables
should be compiled using the normal, unmodified MPI compiler. should be compiled using the normal, unmodified MPI compiler.
...@@ -245,7 +263,7 @@ mpiexec -n 4 -env LD_PRELOAD /home/carns/darshan-install/lib/libdarshan.so mpi-i ...@@ -245,7 +263,7 @@ mpiexec -n 4 -env LD_PRELOAD /home/carns/darshan-install/lib/libdarshan.so mpi-i
srun -n 4 --export=LD_PRELOAD=/home/carns/darshan-install/lib/libdarshan.so mpi-io-test srun -n 4 --export=LD_PRELOAD=/home/carns/darshan-install/lib/libdarshan.so mpi-io-test
---- ----
For sequential programs, the following will set LD_PRELOAD for process duration only: For sequential invocations of MPI programs, the following will set LD_PRELOAD for process duration only:
---- ----
env LD_PRELOAD=/home/carns/darshan-install/lib/libdarshan.so mpi-io-test env LD_PRELOAD=/home/carns/darshan-install/lib/libdarshan.so mpi-io-test
...@@ -288,6 +306,35 @@ exclusively by the executable itself. You can check the rpath of the ...@@ -288,6 +306,35 @@ exclusively by the executable itself. You can check the rpath of the
darshan library by running `objdump -x darshan library by running `objdump -x
/home/carns/darshan-install/lib/libdarshan.so |grep RPATH`. /home/carns/darshan-install/lib/libdarshan.so |grep RPATH`.
== Instrumenting dynamically-linked non-MPI applications
Similar to the process described in the previous section, Darshan relies on the
`LD_PRELOAD` mechanism for instrumenting dynamically-linked non-MPI applications.
This allows Darshan to instrument dynamically-linked binaries produced by non-MPI
compilers (e.g., gcc or clang), extending Darshan instrumentation to new contexts
(like instrumentation of arbitrary Python programs or instrumenting serial
file transfer utilities like `cp` and `scp`).
The only additional step required of Darshan non-MPI users is to also set the
DARSHAN_ENABLE_NONMPI environment variable to signal to Darshan that non-MPI
instrumentation is requested:
----
export DARSHAN_ENABLE_NONMPI=1
----
As described in the previous section, it may be desirable to users to limit the
scope of Darshan's instrumentation by only enabling LD_PRELOAD on the target
executable:
----
env LD_PRELOAD=/home/carns/darshan-install/lib/libdarshan.so io-test
----
[NOTE]
Recall that Darshan instrumentation of non-MPI applications is only possible with
dynamically-linked applications.
== Darshan installation recipes == Darshan installation recipes
The following recipes provide examples for prominent HPC systems. The following recipes provide examples for prominent HPC systems.
...@@ -557,6 +604,7 @@ behavior at runtime: ...@@ -557,6 +604,7 @@ behavior at runtime:
* DARSHAN_MMAP_LOGPATH: if Darshan's mmap log file mechanism is enabled, this variable specifies what path the mmap log files should be stored in (if not specified, log files will be stored in `/tmp`). * DARSHAN_MMAP_LOGPATH: if Darshan's mmap log file mechanism is enabled, this variable specifies what path the mmap log files should be stored in (if not specified, log files will be stored in `/tmp`).
* DARSHAN_EXCLUDE_DIRS: specifies a list of comma-separated paths that Darshan will not instrument at runtime (in addition to Darshan's default blacklist) * DARSHAN_EXCLUDE_DIRS: specifies a list of comma-separated paths that Darshan will not instrument at runtime (in addition to Darshan's default blacklist)
* DXT_ENABLE_IO_TRACE: setting this environment variable enables the DXT (Darshan eXtended Tracing) modules at runtime. Users can specify a numeric value for this variable to set the number of MiB to use for tracing per process; if no value is specified, Darshan will use a default value of 4 MiB. * DXT_ENABLE_IO_TRACE: setting this environment variable enables the DXT (Darshan eXtended Tracing) modules at runtime. Users can specify a numeric value for this variable to set the number of MiB to use for tracing per process; if no value is specified, Darshan will use a default value of 4 MiB.
* DARSHAN_ENABLE_NONMPI: setting this environment variable is required to generate Darshan logs for non-MPI applications
== Debugging == Debugging
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment