Skip to content

GitLab

Commit 5de67837 authored by Jakob Luettgau's avatar Jakob Luettgau
Browse files

Update jupyter notebooks to reflect refactorings (moving plots to... 

Update jupyter notebooks to reflect refactorings (moving plots to experimental, seperating namespaces)
parent 42a6f6de
Expand all Hide whitespace changes
Inline Side-by-side
darshan-util/pydarshan/examples/notebook-darshan-default-object-interface.ipynb → darshan-util/pydarshan/examples/01_darshan-introduction.ipynb
View file @ 5de67837
%% Cell type:code id: tags:
``` python
import darshan
import pprint
```
%% Cell type:code id: tags:
``` python
report = darshan.DarshanReport("example.darshan")
```
%% Cell type:markdown id: tags:
By default only metadata, available modules and the name records are loaded:
%% Cell type:code id: tags:
``` python
report.data
```
%%%% Output: execute_result
{'version': 1,
'records': {},
'metadata': {'job': {'jobid': 4478544,
'uid': 69615,
'start_time': 1490000867,
'end_time': 1490000983,
'metadata': {'lib_ver': '3.1.3', 'h': 'romio_no_indep_rw=true;cb_nodes=4'}},
'exe': '/global/project/projectdirs/m888/glock/tokio-abc-results/bin.edison/vpicio_uni /scratch2/scratchdirs/glock/tokioabc-s.4478544/vpicio/vpicio.hdf5 32',
'mounts': [('/.shared/base/default/etc/dat.conf', 'dvs'),
('/usr/lib64/libibverbs.so.1.0.0', 'dvs'),
('/usr/lib64/libibumad.so.3.0.2', 'dvs'),
('/usr/lib64/librdmacm.so.1.0.0', 'dvs'),
('/usr/lib64/libibgni.so.1.0.0', 'dvs'),
('/global/cscratch1', 'lustre'),
('/global/projectb', 'dvs'),
('/global/projecta', 'dvs'),
('/usr/sbin/ibstat', 'dvs'),
('/global/project', 'dvs'),
('/global/common', 'dvs'),
('/global/syscom', 'dvs'),
('/global/dna', 'dvs'),
('/opt/slurm', 'dvs'),
('/global/u1', 'dvs'),
('/global/u2', 'dvs'),
('/scratch1', 'lustre'),
('/scratch2', 'lustre'),
('/scratch3', 'lustre'),
('/etc', 'dvs'),
('/', 'rootfs'),
('/', 'dvs')],
'modules': {'POSIX': {'len': 186, 'ver': 3, 'idx': 1},
'MPI-IO': {'len': 154, 'ver': 2, 'idx': 2},
'LUSTRE': {'len': 87, 'ver': 1, 'idx': 6},
'STDIO': {'len': 3234, 'ver': 1, 'idx': 7}}},
('/', 'dvs')]},
'modules': {'POSIX': {'len': 186, 'ver': 3, 'idx': 1},
'MPI-IO': {'len': 154, 'ver': 2, 'idx': 2},
'LUSTRE': {'len': 87, 'ver': 1, 'idx': 6},
'STDIO': {'len': 3234, 'ver': 1, 'idx': 7}},
'name_records': {14734109647742566553: '<STDIN>',
15920181672442173319: '<STDOUT>',
7238257241479193519: '<STDERR>',
6301063301082038805: '/scratch2/scratchdirs/glock/tokioabc-s.4478544/vpicio/vpicio.hdf5'}}
%% Cell type:markdown id: tags:
### Loading Additional Records
Additional records can be loaded for example on a per module basis:
%% Cell type:code id: tags:
``` python
# expected to fail
report.data['records']['POSIX']
```
%%%% Output: error
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-8-1c44728d67db> in <module>
<ipython-input-36-1c44728d67db> in <module>
1 # expected to fail
----> 2 report.data['records']['POSIX']
KeyError: 'POSIX'
%% Cell type:code id: tags:
``` python
report.mod_read_all_records("POSIX")
```
%%%% Output: error
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-9-e0928bbeb0bc> in <module>
----> 1 report.mod_read_all_records("POSIX")
~/ANL/darshan-decaf/darshan-workflow/python-packages/darshan/report.py in mod_read_all_records(self, mod, mode)
170 fcn = backend.fcounter_names(mod)
171
--> 172 self.data['modules'][mod]['counters'] = cn
173 self.data['modules'][mod]['fcounters'] = fcn
174 self.data['modules'][mod]['num_records'] = 0
KeyError: 'modules'
%% Cell type:code id: tags:
``` python
report.data['records']['POSIX']
```
%%%% Output: execute_result
[]
[{'id': 6301063301082038805,
'rank': -1,
'counters': array([ 2049, 18446744073709551615, 18446744073709551615,
0, 16402, 16404,
0, 0, 0,
0, 18446744073709551615, 18446744073709551615,
0, 0, 0,
2199023259968, 0, 2199023261831,
0, 0, 0,
16384, 0, 0,
8, 16401, 1048576,
0, 134217728, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
4, 14, 0,
0, 0, 0,
0, 0, 16384,
0, 274743689216, 274743691264,
0, 0, 10240,
4096, 0, 0,
134217728, 272, 544,
328, 16384, 8,
2], dtype=uint64),
'fcounters': array([ 2.04900000e+03, -1.00000000e+00, -1.00000000e+00, 0.00000000e+00,
1.64020000e+04, 1.64040000e+04, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, -1.00000000e+00, -1.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 2.19902326e+12,
0.00000000e+00, 2.19902326e+12, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 1.63840000e+04, 0.00000000e+00, 0.00000000e+00,
8.00000000e+00, 1.64010000e+04, 1.04857600e+06, 0.00000000e+00,
1.34217728e+08, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 4.00000000e+00,
1.40000000e+01, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.63840000e+04,
0.00000000e+00, 2.74743689e+11, 2.74743691e+11, 0.00000000e+00,
0.00000000e+00, 1.02400000e+04, 4.09600000e+03, 0.00000000e+00,
0.00000000e+00, 1.34217728e+08, 2.72000000e+02, 5.44000000e+02,
3.28000000e+02, 1.63840000e+04, 8.00000000e+00, 2.00000000e+00])}]
%% Cell type:markdown id: tags:
### Aggregation (Experimental)
Darshan log data is routinely aggregated for quick overview. The report object offers a few methods to perform common aggregations:
%% Cell type:code id: tags:
``` python
report = darshan.DarshanReport("example.darshan")
```
%% Cell type:markdown id: tags:
Report aggregations and summarization remains experimental for now, mostly to allow interfaces to stabilize. Experimental features can be switched on easily by invoking:
%% Cell type:code id: tags:
``` python
darshan.enable_experimental()
```
%%%% Output: stream
Added method create_time_summary to DarshanReport.
Added method summarize to DarshanReport.
Added method create_timeline to DarshanReport.
Added method agg_ioops to DarshanReport.
Added method create_sankey to DarshanReport.
Added method mod_agg_iohist to DarshanReport.
%% Cell type:code id: tags:
``` python
# expected to fail
report.data['agg_ioops']
```
%%%% Output: error
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-11-20253727a280> in <module>
<ipython-input-41-dc5f73bddcf3> in <module>
1 # expected to fail
----> 2 report.report['agg_ioops']
----> 2 report.data['agg_ioops']
KeyError: 'agg_ioops'
%% Cell type:code id: tags:
``` python
report.mod_read_all_records("POSIX")
report.summarize()
```
%% Cell type:code id: tags:
``` python
# attaches among other summarizes the aggregatedioo
# attaches among other summarizes the aggregated ioops
report.data['agg_ioops']
```
%%%% Output: execute_result
{'POSIX': {'POSIX_OPENS': 2049,
'POSIX_FILENOS': 18446744073709551615,
'POSIX_DUPS': 18446744073709551615,
'POSIX_READS': 0,
'POSIX_WRITES': 16402,
'POSIX_SEEKS': 16404,
'POSIX_STATS': 0,
'POSIX_MMAPS': 0,
'POSIX_FSYNCS': 0,
'POSIX_FDSYNCS': 0,
'POSIX_RENAME_SOURCES': 18446744073709551615,
'POSIX_RENAME_TARGETS': 18446744073709551615,
'POSIX_RENAMED_FROM': 0,
'POSIX_MODE': 0,
'POSIX_BYTES_READ': 0,
'POSIX_BYTES_WRITTEN': 2199023259968,
'POSIX_MAX_BYTE_READ': 0,
'POSIX_MAX_BYTE_WRITTEN': 2199023261831,
'POSIX_CONSEC_READS': 0,
'POSIX_CONSEC_WRITES': 0,
'POSIX_SEQ_READS': 0,
'POSIX_SEQ_WRITES': 16384,
'POSIX_RW_SWITCHES': 0,
'POSIX_MEM_NOT_ALIGNED': 0,
'POSIX_MEM_ALIGNMENT': 8,
'POSIX_FILE_NOT_ALIGNED': 16401,
'POSIX_FILE_ALIGNMENT': 1048576,
'POSIX_MAX_READ_TIME_SIZE': 0,
'POSIX_MAX_WRITE_TIME_SIZE': 134217728,
'POSIX_SIZE_READ_0_100': 0,
'POSIX_SIZE_READ_100_1K': 0,
'POSIX_SIZE_READ_1K_10K': 0,
'POSIX_SIZE_READ_10K_100K': 0,
'POSIX_SIZE_READ_100K_1M': 0,
'POSIX_SIZE_READ_1M_4M': 0,
'POSIX_SIZE_READ_4M_10M': 0,
'POSIX_SIZE_READ_10M_100M': 0,
'POSIX_SIZE_READ_100M_1G': 0,
'POSIX_SIZE_READ_1G_PLUS': 0,
'POSIX_SIZE_WRITE_0_100': 4,
'POSIX_SIZE_WRITE_100_1K': 14,
'POSIX_SIZE_WRITE_1K_10K': 0,
'POSIX_SIZE_WRITE_10K_100K': 0,
'POSIX_SIZE_WRITE_100K_1M': 0,
'POSIX_SIZE_WRITE_1M_4M': 0,
'POSIX_SIZE_WRITE_4M_10M': 0,
'POSIX_SIZE_WRITE_10M_100M': 0,
'POSIX_SIZE_WRITE_100M_1G': 16384,
'POSIX_SIZE_WRITE_1G_PLUS': 0,
'POSIX_STRIDE1_STRIDE': 274743689216,
'POSIX_STRIDE2_STRIDE': 274743691264,
'POSIX_STRIDE3_STRIDE': 0,
'POSIX_STRIDE4_STRIDE': 0,
'POSIX_STRIDE1_COUNT': 10240,
'POSIX_STRIDE2_COUNT': 4096,
'POSIX_STRIDE3_COUNT': 0,
'POSIX_STRIDE4_COUNT': 0,
'POSIX_ACCESS1_ACCESS': 134217728,
'POSIX_ACCESS2_ACCESS': 272,
'POSIX_ACCESS3_ACCESS': 544,
'POSIX_ACCESS4_ACCESS': 328,
'POSIX_ACCESS1_COUNT': 16384,
'POSIX_ACCESS2_COUNT': 8,
'POSIX_ACCESS3_COUNT': 2},
'POSIX_simple': {'Read': 0,
'Write': 16402,
'Open': 2049,
'Stat': 0,
'Seek': 16404,
'Mmap': 0,
'Fsync': 0}}
%% Cell type:markdown id: tags:
Or fine grained:
%% Cell type:code id: tags:
``` python
report.mod_agg_iohist("MPI-IO") # to create the histograms
```
%% Cell type:code id: tags:
``` python
report.agg_ioops() # to create the combined operation type summary
```
%% Cell type:markdown id: tags:
### DXT Records
DXT records are also supported, and can be loaded individually on a per module basis as follows:
%% Cell type:code id: tags:
``` python
report2 = darshan.DarshanReport("example.darshan")
report2.mod_read_all_dxt_records("ABC")
```
%%%% Output: stream
Skipping. Log does not contain data for mod: ABC
%% Cell type:code id: tags:
``` python
report2.data
```
%%%% Output: error
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-4-b6301753ed15> in <module>
----> 1 report2.data
%%%% Output: execute_result
NameError: name 'report2' is not defined
{'version': 1,
'records': {},
'metadata': {'job': {'jobid': 4478544,
'uid': 69615,
'start_time': 1490000867,
'end_time': 1490000983,
'metadata': {'lib_ver': '3.1.3', 'h': 'romio_no_indep_rw=true;cb_nodes=4'}},
'exe': '/global/project/projectdirs/m888/glock/tokio-abc-results/bin.edison/vpicio_uni /scratch2/scratchdirs/glock/tokioabc-s.4478544/vpicio/vpicio.hdf5 32',
'mounts': [('/.shared/base/default/etc/dat.conf', 'dvs'),
('/usr/lib64/libibverbs.so.1.0.0', 'dvs'),
('/usr/lib64/libibumad.so.3.0.2', 'dvs'),
('/usr/lib64/librdmacm.so.1.0.0', 'dvs'),
('/usr/lib64/libibgni.so.1.0.0', 'dvs'),
('/global/cscratch1', 'lustre'),
('/global/projectb', 'dvs'),
('/global/projecta', 'dvs'),
('/usr/sbin/ibstat', 'dvs'),
('/global/project', 'dvs'),
('/global/common', 'dvs'),
('/global/syscom', 'dvs'),
('/global/dna', 'dvs'),
('/opt/slurm', 'dvs'),
('/global/u1', 'dvs'),
('/global/u2', 'dvs'),
('/scratch1', 'lustre'),
('/scratch2', 'lustre'),
('/scratch3', 'lustre'),
('/etc', 'dvs'),
('/', 'rootfs'),
('/', 'dvs')]},
'modules': {'POSIX': {'len': 186, 'ver': 3, 'idx': 1},
'MPI-IO': {'len': 154, 'ver': 2, 'idx': 2},
'LUSTRE': {'len': 87, 'ver': 1, 'idx': 6},
'STDIO': {'len': 3234, 'ver': 1, 'idx': 7}},
'name_records': {14734109647742566553: '<STDIN>',
15920181672442173319: '<STDOUT>',
7238257241479193519: '<STDERR>',
6301063301082038805: '/scratch2/scratchdirs/glock/tokioabc-s.4478544/vpicio/vpicio.hdf5'}}
%% Cell type:markdown id: tags:
### Report Algebra
By overloading the addition operator, reports can be easily merge for a variety of counters.
Some choices to make though
%% Cell type:code id: tags:
``` python
report1 = darshan.DarshanReport("example.darshan")
report2 = darshan.DarshanReport("example.darshan")
combined = report1 + report2
r1 = darshan.DarshanReport("example.darshan")
r2 = darshan.DarshanReport("example2.darshan")
combined = r1 + r2
combined
```
%%%% Output: execute_result
<darshan.report.DarshanReport at 0x7faa94613e10>
%% Cell type:markdown id: tags:
### Error Handling?
Currently, playing with two modes, both have their pros and cons.
Generally, should expose errors and let users handle them. At the same time, just skipping invalid load requests does little harm but greatly improves convienince.
Could add a switch to enable disable these guard :/
%% Cell type:code id: tags:
``` python
report.mod_read_all_records("ABC")
```
%%%% Output: error
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-49-476a7dad602f> in <module>
----> 1 report.mod_read_all_records("ABC")
~/ANL/darshan-decaf/darshan-workflow/python-packages/darshan/report.py in mod_read_all_records(self, mod, mode)
185 fcn = backend.fcounter_names(mod)
186
--> 187 self.data['modules'][mod]['counters'] = cn
188 self.data['modules'][mod]['fcounters'] = fcn
189 self.data['modules'][mod]['num_records'] = 0
KeyError: 'ABC'
%% Cell type:code id: tags:
``` python
report.mod_read_all_dxt_records("ABC")
```
%%%% Output: stream
Skipping. Log does not contain data for mod: ABC
%% Cell type:code id: tags:
``` python
```
......
darshan-util/pydarshan/examples/03_darshan-report-algebra.ipynb 0 → 100644
View file @ 5de67837
%% Cell type:code id: tags:
``` python
import darshan
import pprint
```
%% Cell type:markdown id: tags:
### Report Algebra
By overloading the addition operator, reports can be easily merge for a variety of counters.
Some choices to make though
%% Cell type:code id: tags:
``` python
r1 = darshan.DarshanReport("example.darshan")
r2 = darshan.DarshanReport("example2.darshan")
```
%% Cell type:code id: tags:
``` python
r1.data
```
%%%% Output: execute_result
{'version': 1,
'records': {},
'metadata': {'job': {'jobid': 4478544,
'uid': 69615,
'start_time': 1490000867,
'end_time': 1490000983,
'metadata': {'lib_ver': '3.1.3', 'h': 'romio_no_indep_rw=true;cb_nodes=4'}},
'exe': '/global/project/projectdirs/m888/glock/tokio-abc-results/bin.edison/vpicio_uni /scratch2/scratchdirs/glock/tokioabc-s.4478544/vpicio/vpicio.hdf5 32',
'mounts': [('/.shared/base/default/etc/dat.conf', 'dvs'),
('/usr/lib64/libibverbs.so.1.0.0', 'dvs'),
('/usr/lib64/libibumad.so.3.0.2', 'dvs'),
('/usr/lib64/librdmacm.so.1.0.0', 'dvs'),
('/usr/lib64/libibgni.so.1.0.0', 'dvs'),
('/global/cscratch1', 'lustre'),
('/global/projectb', 'dvs'),
('/global/projecta', 'dvs'),
('/usr/sbin/ibstat', 'dvs'),
('/global/project', 'dvs'),
('/global/common', 'dvs'),
('/global/syscom', 'dvs'),
('/global/dna', 'dvs'),
('/opt/slurm', 'dvs'),
('/global/u1', 'dvs'),
('/global/u2', 'dvs'),
('/scratch1', 'lustre'),
('/scratch2', 'lustre'),
('/scratch3', 'lustre'),
('/etc', 'dvs'),
('/', 'rootfs'),
('/', 'dvs')]},
'modules': {'POSIX': {'len': 186, 'ver': 3, 'idx': 1},
'MPI-IO': {'len': 154, 'ver': 2, 'idx': 2},
'LUSTRE': {'len': 87, 'ver': 1, 'idx': 6},
'STDIO': {'len': 3234, 'ver': 1, 'idx': 7}},
'name_records': {14734109647742566553: '<STDIN>',
15920181672442173319: '<STDOUT>',
7238257241479193519: '<STDERR>',
6301063301082038805: '/scratch2/scratchdirs/glock/tokioabc-s.4478544/vpicio/vpicio.hdf5'}}
%% Cell type:code id: tags:
``` python
r1.data
```
%%%% Output: execute_result
{'version': 1,
'records': {},
'metadata': {'job': {'jobid': 4478544,
'uid': 69615,
'start_time': 1490000867,
'end_time': 1490000983,
'metadata': {'lib_ver': '3.1.3', 'h': 'romio_no_indep_rw=true;cb_nodes=4'}},
'exe': '/global/project/projectdirs/m888/glock/tokio-abc-results/bin.edison/vpicio_uni /scratch2/scratchdirs/glock/tokioabc-s.4478544/vpicio/vpicio.hdf5 32',
'mounts': [('/.shared/base/default/etc/dat.conf', 'dvs'),
('/usr/lib64/libibverbs.so.1.0.0', 'dvs'),
('/usr/lib64/libibumad.so.3.0.2', 'dvs'),
('/usr/lib64/librdmacm.so.1.0.0', 'dvs'),
('/usr/lib64/libibgni.so.1.0.0', 'dvs'),
('/global/cscratch1', 'lustre'),
('/global/projectb', 'dvs'),
('/global/projecta', 'dvs'),
('/usr/sbin/ibstat', 'dvs'),
('/global/project', 'dvs'),
('/global/common', 'dvs'),
('/global/syscom', 'dvs'),
('/global/dna', 'dvs'),
('/opt/slurm', 'dvs'),
('/global/u1', 'dvs'),
('/global/u2', 'dvs'),
('/scratch1', 'lustre'),
('/scratch2', 'lustre'),
('/scratch3', 'lustre'),
('/etc', 'dvs'),
('/', 'rootfs'),
('/', 'dvs')]},
'modules': {'POSIX': {'len': 186, 'ver': 3, 'idx': 1},
'MPI-IO': {'len': 154, 'ver': 2, 'idx': 2},
'LUSTRE': {'len': 87, 'ver': 1, 'idx': 6},
'STDIO': {'len': 3234, 'ver': 1, 'idx': 7}},
'name_records': {14734109647742566553: '<STDIN>',
15920181672442173319: '<STDOUT>',
7238257241479193519: '<STDERR>',
6301063301082038805: '/scratch2/scratchdirs/glock/tokioabc-s.4478544/vpicio/vpicio.hdf5'}}
%% Cell type:code id: tags:
``` python
r1.mod_read_all_records("POSIX")
len(r1.data['records']['POSIX'])
```
%%%% Output: execute_result
1
%% Cell type:code id: tags:
``` python
r2.mod_read_all_records("MPI-IO")
len(r2.data['records']['MPI-IO'])
```
%%%% Output: execute_result
1
%% Cell type:code id: tags:
``` python
r1.data['records']['POSIX']
```
%%%% Output: execute_result
[{'id': 6301063301082038805,
'rank': -1,
'counters': array([ 2049, 18446744073709551615, 18446744073709551615,
0, 16402, 16404,
0, 0, 0,