Commit 9c4ed094 authored by Jakob Luettgau's avatar Jakob Luettgau
Browse files

Update notebooks and part of the documention.

parent 94d3a016
......@@ -6,7 +6,7 @@ SPHINXOPTS =
SPHINXBUILD = python -m sphinx
SPHINXPROJ = pydarshan
SOURCEDIR = .
BUILDDIR = _build
BUILDDIR = build
# Put it first so that "make" without argument is like "make help".
help:
......
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# pydarshan documentation build configuration file, created by
# Documentation build configuration file, created by
# sphinx-quickstart on Fri Jun 9 13:47:02 2017.
#
# This file is execfile()d with the current directory set to its
......@@ -63,7 +63,7 @@ source_suffix = '.rst'
master_doc = 'index'
# General information about the project.
project = u'darshan'
project = u'Darshan'
copyright = u"2019, ANL"
author = u""
......@@ -128,9 +128,7 @@ htmlhelp_basename = 'darshandoc'
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'pydarshan',
u'pydarshan Documentation',
[author], 1)
(master_doc, 'pydarshan', u'pydarshan Documentation', [author], 1)
]
......@@ -159,9 +157,7 @@ latex_elements = {
# (source start file, target name, title, author, documentclass
# [howto, manual, or own class]).
latex_documents = [
(master_doc, 'pydarshan.tex',
u'pydarshan Documentation',
u'Argonne National Laboratory', 'manual'),
(master_doc, 'pydarshan.tex', u'pydarshan Documentation', u'Argonne National Laboratory', 'manual'),
]
......@@ -171,12 +167,7 @@ latex_documents = [
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'pydarshan',
u'pydarshan Documentation',
author,
'pydarshan',
'One line description of project.',
'Miscellaneous'),
(master_doc, 'pydarshan', u'pydarshan Documentation', author, 'pydarshan', 'Python utilities for Darshan HPC I/O instrumentation.', 'Miscellaneous'),
]
......
......@@ -4,14 +4,6 @@ darshan.backend package
Submodules
----------
darshan.backend.ascii\_backend module
-------------------------------------
.. automodule:: darshan.backend.ascii_backend
:members:
:undoc-members:
:show-inheritance:
darshan.backend.cffi\_backend module
------------------------------------
......
......@@ -7,6 +7,7 @@ Subpackages
.. toctree::
darshan.backend
darshan.cli
darshan.experimental
Submodules
......
......@@ -8,7 +8,7 @@ This documenation is only for the Darshan Python bindings, for documentation for
:caption: Contents:
readme
installation
setup
usage
.. toctree::
......
.. highlight:: shell
============
Installation
============
Stable release
--------------
To install pydarshan, run this command in your terminal:
.. code-block:: console
$ pip install pydarshan
This is the preferred method to install pydarshan, as it will always install the most recent stable release.
If you don't have `pip`_ installed, this `Python installation guide`_ can guide
you through the process.
.. _pip: https://pip.pypa.io
.. _Python installation guide: https://www.mcs.anl.gov/research/projects/darshan/
Pydarshan assumes that a recent 'darshan-utils' is installed as a shared
library. If darshan-util is not installed consult with the darshan
documentation or consider using `Spack`_ to install::
spack install darshan-util
.. _Spack: https://spack.io/
From sources
------------
The sources for pydarshan can be downloaded from the `Github repo`_.
You can either clone the public repository:
.. code-block:: console
$ git clone https://xgitlab.cels.anl.gov/darshan/darshan.git
$ cd darshan/darshan-util/pydarshan
.. code-block:: console
$ python setup.py install
.. _Github repo: https://xgitlab.cels.anl.gov/darshan/darshan
%% Cell type:markdown id: tags:
# DarshanUtils for Python
This notebook gives an overwiew of features provided by the Python bindings for DarshanUtils.
%% Cell type:markdown id: tags:
By default all records, metadata, available modules and the name records are loaded:
%% Cell type:code id: tags:
``` python
import darshan
report = darshan.DarshanReport("example.darshan", read_all=True) # Default behavior
report = darshan.DarshanReport("example-logs/example.darshan", read_all=True) # Default behavior
report.info()
```
%% Output
Filename: example.darshan
Filename: example-logs/example.darshan
Times: 2017-03-20 10:07:47 to 2017-03-20 10:09:43 (Duration 0:01:56)
Executeable: /global/project/projectdirs/m888/glock/tokio-abc-results/bin.edison/vpicio_uni /scratch2/scratchdirs/glock/tokioabc-s.4478544/vpicio/vpicio.hdf5 32
Processes: 2048
JobID: 4478544
UID: 69615
Modules in Log: ['POSIX', 'MPI-IO', 'LUSTRE', 'STDIO']
Loaded Records: {'POSIX': 1, 'MPI-IO': 1, 'STDIO': 129}
Name Records: 4
Darshan/Hints: {'lib_ver': '3.1.3', 'h': 'romio_no_indep_rw=true;cb_nodes=4'}
DarshanReport: id(140525897881584) (tmp)
DarshanReport: id(139721554559216) (tmp)
%% Cell type:code id: tags:
``` python
report.modules
```
%% Output
{'POSIX': {'len': 186, 'ver': 3, 'idx': 1, 'num_records': 1},
'MPI-IO': {'len': 154, 'ver': 2, 'idx': 2, 'num_records': 1},
'LUSTRE': {'len': 87, 'ver': 1, 'idx': 7},
'STDIO': {'len': 3234, 'ver': 1, 'idx': 8, 'num_records': 129}}
%% Cell type:code id: tags:
``` python
```
%% Cell type:code id: tags:
``` python
report.mod_records
```
%% Output
<bound method DarshanReport.mod_records of <darshan.report.DarshanReport object at 0x7fcebc3997f0>>
<bound method DarshanReport.mod_records of <darshan.report.DarshanReport object at 0x7f1375a080f0>>
%% Cell type:markdown id: tags:
A few of the internal data structures explained:
%% Cell type:code id: tags:
``` python
# report.metadata # dictionary with raw metadata from darshan log
# report.modules # dictionary with raw module info from darshan log (need: technical, module idx)
# report.name_records # dictionary for resovling name records: id -> path/name
# report.records # per module "dataframes"/dictionaries holding loaded records
```
%% Cell type:markdown id: tags:
The darshan report holds a variety of namespaces for report related data. All of them are also referenced in `report.data` at the moment, but reliance on this internal organization of the report object is discouraged once the API stabilized. Currently, `report.data` references the following information:
%% Cell type:code id: tags:
``` python
report.data.keys()
```
%% Output
dict_keys(['version', 'metadata', 'records', 'summary', 'modules', 'counters', 'name_records', 'mounts'])
%% Cell type:code id: tags:
``` python
report.mod_read_all_records('POSIX')
```
%% Cell type:code id: tags:
``` python
report.mod_read_all_records('STDIO')
```
%% Cell type:code id: tags:
``` python
report.update_name_records()
report.info()
```
%% Output
POSIX
MPI-IO
STDIO
Filename: example.darshan
Filename: example-logs/example.darshan
Times: 2017-03-20 10:07:47 to 2017-03-20 10:09:43 (Duration 0:01:56)
Executeable: /global/project/projectdirs/m888/glock/tokio-abc-results/bin.edison/vpicio_uni /scratch2/scratchdirs/glock/tokioabc-s.4478544/vpicio/vpicio.hdf5 32
Processes: 2048
JobID: 4478544
UID: 69615
Modules in Log: ['POSIX', 'MPI-IO', 'LUSTRE', 'STDIO']
Loaded Records: {'POSIX': 1, 'MPI-IO': 1, 'STDIO': 129}
Name Records: 3
Darshan/Hints: {'lib_ver': '3.1.3', 'h': 'romio_no_indep_rw=true;cb_nodes=4'}
DarshanReport: id(140525897881584) (tmp)
DarshanReport: id(139721554559216) (tmp)
%% Cell type:code id: tags:
``` python
from IPython.display import display, HTML
# use: display(obj)
```
%% Cell type:markdown id: tags:
### Selectively Loading Records
For memory efficiant analysis, it is possible to supress records from being loaded automatically. This is useful, for example, when analysis considers only records of a particular layer/module.