Individual operations in the DUMPI traces can be inspected by using the codes-workload-dump utility (in src/workload). This utility provides detailed information on each of the MPI operations in the logs e.g. operation type (whether send/recv/wait/collective), tag number, number of bytes transmitted (derived from the data type and count of the MPI operation), start and end times of operation.
Note that running this utility on a large number of traces will generate a lot of output so it is preferred to run it on a small number of MPI ranks (<4). Using the -s command line option will display the aggregate statistics.
./bin/codes-workload-dump --type "dumpi-trace-workload" -s --num-ranks n --dumpi-log dumpi-trace-dir/dumpi-2014.03.06.20.34.26-
Note on trace reading - the input file prefix to the dumpi workload generator
should be everything up to the rank number. E.g., if the dumpi files are of the
form "dumpi-YYYY.MM.DD.HH.MM.SS-XXXX.bin", then the input should be
Replaying Application Trace on CODES MPI Simulation Layer
Download and untar the DUMPI AMG application trace for 216 MPI ranks using the following download link:
The simulation runs in ROSS serial, conservative and optimistic modes. Note: Dragonfly and torus networks may have more number of nodes in the network than the number network traces (Some network nodes will only forward packets and they will not end up loading the traces). Thats why --num_net_traces argument is used to specify exact number of traces available in the DUMPI directory if there is a mis-match between number of network nodes and traces.* Running the simulation in optimistic mode
## Replaying Multiple Application Traces on Network Models* Generate job allocation file (random or contiguous) using python scripts.Allocation options 1. Random allocation assigns a set of randomly selected network nodes to each job. 1. Contiguous allocation assigns a set of contiguous network nodes to the jobs. See [generation CODES allocations](https://xgitlab.cels.anl.gov/codes/codes/wikis/codes-allocations) for instructions on how to generate joballocation file using python. Example allocation files are insrc/network-workloads/conf/allocation-rand.conf, allocation-cont.conf.* Run the simulation with multiple job allocations
## Running multiple MPI ranks (or cores) mapped to a compute node 13- Update the config file to have multiple core/proc LPs mapped on to amodel-net LP. See example config file in:`src/network-workloads/conf/modelnet-mpi-test-dfly-mul-cores.conf`The overhead to send a message to the same node can be configured through the self_msg_overhead parameter in the config file (units are nanosecs). Its default value is 10 ns.14- If running multiple MPI ranks per node with random allocations, theallocation files must also be generated with multiple cores per node. See[generating job allocations](https://xgitlab.cels.anl.gov/codes/codes/wikis/codes-allocations) for how to generate allocation files that usemultiple cores per node.## Sampling and debugging options Runtime options can be used to enable time-stepped series data of simulationwith multiple workloads:--**enable_sampling = 1** [Enables sampling of network & workload statistics after a specific simulated interval.Default sampling interval is 5 millisec and default sampling end time is 3secs. These values can be adjusted at runtime using --sampling_interval and--sampling_end_time options.]--**lp-io-dir-dir-name** [Turns on end of simulation statistics for dragonfly network model]**NOTE: Each network model has information on how to run it with DUMPI Traces. For details, see [CODES network models](https://xgitlab.cels.anl.gov/codes/codes/wikis/codes-networks)**