Skip to content

GitLab

Commit 9bd64e7d authored by Kevin Harms's avatar Kevin Harms
Browse files

Set of test cases for checking performance estimation of darshan-parser 

--perf.


git-svn-id: https://svn.mcs.anl.gov/repos/darshan/trunk@385 3b7491f3-a168-0410-bf4b-c445ed680a29
parent 7e8fa01c
Hide whitespace changes
Inline Side-by-side
Showing with 250 additions and 0 deletions
test/perf/350665.output 0 → 100644
View file @ 9bd64e7d
IOR-2.10.0.1: MPI Coordinated Test of Parallel I/O
Run began: Tue Feb 8 21:50:51 2011
Command line used: /home/harms/bgp/IOR/src/C/IOR -a POSIX -F -b 256M -g -m -i 4 -K -o /intrepid-fs1/users/harms/scratch/ior.out -r -w -t 4m -v
Machine: CNK ion-R42-1
Start time skew across all tasks: 0.20 sec
Path: /intrepid-fs1/users/harms/scratch
FS: 2194.4 TiB Used FS: 22.8% Inodes: 4096.0 Mi Used Inodes: 33.7%
Participating tasks: 2048
Summary:
api = POSIX
test filename = /intrepid-fs1/users/harms/scratch/ior.out
access = file-per-process
pattern = segmented (1 segment)
ordering = sequential offsets
clients = 2048 (1 per node)
repetitions = 4
xfersize = 4 MiB
blocksize = 256 MiB
aggregate filesize = 512 GiB
Commencing write performance test.
Tue Feb 8 21:51:00 2011
access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) iter
------ --------- ---------- --------- -------- -------- -------- ----
write 1853.28 262144 4096 6.46 276.33 0.104290 0
Commencing read performance test.
Tue Feb 8 21:55:54 2011
read 1568.01 262144 4096 6.39 327.85 0.128561 0
Commencing write performance test.
Tue Feb 8 22:02:12 2011
write 1860.19 262144 4096 6.68 275.05 0.108472 1
Commencing read performance test.
Tue Feb 8 22:07:05 2011
read 1497.59 262144 4096 6.39 343.58 0.120167 1
Commencing write performance test.
Tue Feb 8 22:13:36 2011
write 1905.81 262144 4096 4.29 270.71 0.103336 2
Commencing read performance test.
Tue Feb 8 22:18:25 2011
read 1558.55 262144 4096 6.36 329.91 0.126614 2
Commencing write performance test.
Tue Feb 8 22:24:44 2011
write 1952.53 262144 4096 5.20 263.21 0.101522 3
Commencing read performance test.
Tue Feb 8 22:29:25 2011
read 1566.06 262144 4096 6.42 328.24 0.120201 3
Max Write: 1952.53 MiB/sec (2047.38 MB/sec)
Max Read: 1568.01 MiB/sec (1644.17 MB/sec)
Run finished: Tue Feb 8 22:35:34 2011
test/perf/350690.output 0 → 100644
View file @ 9bd64e7d
stdout[0] IOR-2.10.0.1: MPI Coordinated Test of Parallel I/O
stdout[0] Run began: Tue Feb 8 23:10:58 2011
stdout[0] Command line used: /home/harms/bgp/IOR/src/C/IOR -a POSIX -b 256M -g -m -i 4 -K -o /intrepid-fs0/users/harms/scratch/ior.out -r -w -t 4m -v
stdout[0] Machine: CNK ion-R44-9
stdout[0] Start time skew across all tasks: 0.12 sec
stdout[0] Path: /intrepid-fs0/users/harms/scratch
stdout[0] FS: 3072.0 TiB Used FS: 90.3% Inodes: 512.0 Mi Used Inodes: 76.2%
stdout[0] Participating tasks: 2048
stdout[0]
Summary:
stdout[0] api = POSIX
stdout[0] test filename = /intrepid-fs0/users/harms/scratch/ior.out
stdout[0] access = single-shared-file
stdout[0] pattern = segmented (1 segment)
stdout[0] ordering = sequential offsets
stdout[0] clients = 2048 (1 per node)
stdout[0] repetitions = 4
stdout[0] xfersize = 4 MiB
stdout[0] blocksize = 256 MiB
stdout[0] aggregate filesize = 512 GiB
stdout[0]
stdout[0] Commencing write performance test.
stdout[0] Tue Feb 8 23:10:59 2011
stdout[0]
stdout[0] access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) iter
stdout[0] ------ --------- ---------- --------- -------- -------- -------- ----
stdout[0] write 2222.96 262144 4096 0.427417 235.34 0.079745 0
stdout[0] Commencing read performance test.
stdout[0] Tue Feb 8 23:14:54 2011
stdout[0]
stdout[0] read 2878.55 262144 4096 0.017130 182.00 0.123749 0
stdout[0] Commencing write performance test.
stdout[0] Tue Feb 8 23:17:57 2011
stdout[0]
stdout[0] write 2321.09 262144 4096 0.179439 225.64 0.063335 1
stdout[0] Commencing read performance test.
stdout[0] Tue Feb 8 23:21:42 2011
stdout[0]
stdout[0] read 2939.11 262144 4096 0.018604 178.19 0.170030 1
stdout[0] Commencing write performance test.
stdout[0] Tue Feb 8 23:24:41 2011
stdout[0]
stdout[0] write 2314.76 262144 4096 0.183715 226.12 0.194725 2
stdout[0] Commencing read performance test.
stdout[0] Tue Feb 8 23:28:28 2011
stdout[0]
stdout[0] read 2910.44 262144 4096 0.017381 179.94 0.181669 2
stdout[0] Commencing write performance test.
stdout[0] Tue Feb 8 23:31:28 2011
stdout[0]
stdout[0] write 2302.51 262144 4096 0.166151 227.30 0.233268 3
stdout[0] Commencing read performance test.
stdout[0] Tue Feb 8 23:35:16 2011
stdout[0]
stdout[0] read 2903.94 262144 4096 0.017946 180.34 0.189010 3
stdout[0]
stdout[0] Max Write: 2321.09 MiB/sec (2433.84 MB/sec)
stdout[0] Max Read: 2939.11 MiB/sec (3081.88 MB/sec)
stdout[0]
stdout[0] Run finished: Tue Feb 8 23:38:16 2011
test/perf/350892.output 0 → 100644
View file @ 9bd64e7d
IOR-2.10.0.1: MPI Coordinated Test of Parallel I/O
Run began: Wed Feb 9 20:37:39 2011
Command line used: /home/harms/bgp/IOR/src/C/IOR -f single_and_multi_file.ior
Machine: CNK ion-R06-9
Summary:
api = POSIX
test filename = /intrepid-fs0/users/harms/scratch/ior.out
access = single-shared-file
ordering = sequential offsets
clients = 2048 (1 per node)
repetitions = 4
xfersize = 4 MiB
blocksize = 256 MiB
aggregate filesize = 512 GiB
access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) iter
------ --------- ---------- --------- -------- -------- -------- ----
write 2263.37 262144 4096 0.318107 231.21 0.107741 0
read 2819.49 262144 4096 0.016437 185.79 0.145695 0
write 2363.48 262144 4096 0.186718 221.42 0.216908 1
read 2861.37 262144 4096 0.017310 182.96 0.251082 1
write 2261.83 262144 4096 0.167811 231.40 0.234639 2
read 2867.13 262144 4096 0.016909 182.67 0.179075 2
write 2304.19 262144 4096 0.218270 227.20 0.118000 3
read 2908.42 262144 4096 0.018176 180.11 0.134019 3
Max Write: 2363.48 MiB/sec (2478.29 MB/sec)
Max Read: 2908.42 MiB/sec (3049.70 MB/sec)
IOR-2.10.0.1: MPI Coordinated Test of Parallel I/O
Run began: Wed Feb 9 21:05:05 2011
Command line used: /home/harms/bgp/IOR/src/C/IOR -f single_and_multi_file.ior
Machine: CNK ion-R06-9
Summary:
api = POSIX
test filename = /intrepid-fs0/users/harms/scratch/ior.out
access = file-per-process
ordering = sequential offsets
clients = 2048 (1 per node)
repetitions = 4
xfersize = 4 MiB
blocksize = 256 MiB
aggregate filesize = 512 GiB
write 2341.54 262144 4096 0.622082 223.12 0.162660 0
read 2862.17 262144 4096 0.017030 183.13 0.030806 0
write 2354.37 262144 4096 0.409148 222.11 0.171205 1
read 2850.33 262144 4096 0.016691 183.89 0.031446 1
write 2356.01 262144 4096 0.425325 221.91 0.200420 2
read 2867.77 262144 4096 0.020235 182.77 0.032219 2
write 2341.94 262144 4096 0.982192 222.72 0.163251 3
read 2884.45 262144 4096 0.054666 181.67 0.035870 3
Max Write: 2356.01 MiB/sec (2470.45 MB/sec)
Max Read: 2884.45 MiB/sec (3024.56 MB/sec)
Run finished: Wed Feb 9 21:32:21 2011
test/perf/README 0 → 100644
View file @ 9bd64e7d
Test Cases to evaluate darshan-parser --perf performance calculations.
Logs
----
350665
* IOR single shared file with multiple iterations
* 350665.output
* harms_IOR_id350665_2-8-78651_12.darshan.gz
350690
* IOR file-per-process with multiple iterations
* 350690
* harms_IOR_id350690_2-8-83458_1.darshan.gz
350892
* IOR with both single file and file-per-process modes
* 350892.output
* harms_IOR_id1350892_2-9-74258_1.darshan.gz
Check Performance
------------------
Add all the performance test runs (both read and write) and the compute the average.
awk 'BEGIN {sum=0;count=0;} /^(write|read)/ { sum+=$2; count+=1; } END { print "avg: " sum/count; }' xxxx.output
darshan-parser --perf log.gz
Compare the four methods to the calculated average.
IOR Commands
------------
# File Per Proc with Multiple Files
qsub -q prod -A Operations -t 60 -n 512 --mode vn --env BG_COREDUMPDISABLED=1 IOR/src/C/IOR -a POSIX -F -b 256M -g -m -i 4 -K -o <path>/ior.out -r -w -t 4m -v
# Single Shared File with Multiple Files
qsub -q prod -A Operations -t 60 -n 512 --mode vn --env BG_COREDUMPDISABLED=1 IOR/src/C/IOR -a POSIX -b 256M -g -m -i 4 -K -o <path>/ior.out -r -w -t 4m -v
# Both I/O methods in one run
qsub -q prod -A Operations -t 60 -n 512 --mode vn --env BG_COREDUMPDISABLED=1 IOR/src/C/IOR -f single_and_multi_file.ior
test/perf/single_and_multi_file.ior 0 → 100644
View file @ 9bd64e7d
# IOR Script to use both single-shared-file and file-per-proc IO
#
#qsub -q prod -A Operations -t 60 -n 512 --mode vn --env BG_COREDUMPDISABLED=1 /
home/harms/bgp/IOR/src/C/IOR -f file.ior
#
IOR START
# Base Config
testFile = /intrepid-fs0/users/harms/scratch/ior.out
api = POSIX
blockSize = 256M
transferSize = 4M
intraTestBarriers = 1
keepFileWithError = 1
multiFile = 1
repetitions = 4
writeFile = 1
readFile = 1
# Single Shared File
filePerProc = 0
RUN
# File Per Process
filePerProc = 1
uniqueDir = 1
RUN
IOR STOP
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment