Poor Performance on Small Files
Reading from Bebop filesystem [turam@bdw-0319 turam]$ python time_gio.py 03_31_2018.OR.401.coreproperties x Columns of data to read: ['x'] Runtime to read in 1 properties of 7.54e+08 halos = 33.084 seconds Runtime to load the same data from a Numpy binary = 1.462 seconds For fname = `03_31_2018.OR.401.coreproperties`, gio reads 22.63x slower than numpy Reading from /dev/shm: [turam@bdw-0319 shm]$ python time_gio.py 03_31_2018.OR.401.coreproperties x Runtime to read in 1 properties of 7.54e+08 halos = 4.730 seconds Runtime to load the same data from a Numpy binary = 2.116 seconds For fname = `03_31_2018.OR.401.coreproperties`, gio reads 2.24x slower than numpy Using a simple c++ genericio code: mpiexec -n 64 /home/turam/dev/genericio//mpi/gioread 03_31_2018.OR.401.coreproperties x Read 1 variables from 03_31_2018.OR.401.coreproperties (3016146524 bytes) in 0.682227s: 4216.22 MB/s [excluding header read]
- Copy small files into /dev/shm before reading.
- As an optimization, GenericIO can do one large read if reading the whole file.
- Use GenericIORewrite to rewrite them for a small number of ranks (maybe do this as a standard "cleanup" after jobs for small fies. Needs to be careful about correlations between rank blocks between files.
- Implement a GenericIO interface that uses readv to avoid as many file-system syscalls.
- We can also change the subfiles used for smaller files.
- Note: The python interface opens/closes each file a lot. That can be cleaned up.