changing rshift to shift both read and write ops, because we have unchache flag now. The rshift now provides a case where strided agg placement is a bith off

updtate of topology branch - adding uncache option. About to change readshift option to shift ranks for read and write (seems to highlight topology placement importance

improving topology version of exerciser to allow contrived example, where agg selection should matter. maybe

revising documentation, and setting up exerciser to run with CCIO VFDL containing topology-aware aggregator selection

adding option --rshift to require the read to occur on data that is shifted from position where the rank originally wrote to.  By setting this to ppn, the option should allow us to measure the read performance with mitigated caching effects

adding topology api to the exerciser code to test the effect of aggregator selection on the collective performance

Commenting out the USING_CCIO definition in Exerciser/exerciser.c (keeping master branch compatible with official HDF5 distributions)

merging some temporary changes from chunked branch. The chunked storage mode will now be set to H5FD_MPIO_CHUNK_MULTI_IO and only one iteration will be used by default.  In the future, the chunk mode and number of iteration should be adjustable by a command-line argument (like everything else)

Adding --nowrite option to dissable the calls to MPI_File_Write_at inside the CCIO branch of HDF5 -- Note that this option also disables the read and verification tests in the exerciser

Correcting serious issue with BW calculations -- Needed to change the command-line args to read in the number of elements in each dimension of the minimum-size dataset (NOT the number of bytes)

New changes allow the user to set the stride and block size of the data in local memory (of the hyperslab to be written to a file), as well as a list of file-dataspace block and stride sizes (for each dimension in the file).  The local memory buffer is still treated as a 1d array, and the custom hyperslab selection is only used for the double-type dataset (and not the derived-type dataset, yet)

Adding more options to the exerciser (including new options to add custom stride distances, memory hogging, performance counting, and self validation). There is also a new Excerciser/Cray/ directory with an example script and corresponding README.md file (to help Cray get started with this testing code). Note that the memory hoggin option and custom stride options (--memuse and --stride) are somewhat limited and experimental). In this version of the code, the hdf5 datasets must be <=4 dimensions (this is easy to change in the future, but was more convenient for now).

Richard Zamora authored Feb 16, 2018 and Paul Coffman committed Mar 14, 2018

initial changes to replace minbuf -> maxbuf approach with minbuf, numBufLoops, bufMultiple

adding multiple-dimension code... not working yet (this is just a checkpoint)

some fixes, basic multidimensions working

basic multi-dimensional functionality -- need to test/debug for dervived types and attributes -- currently spliting 0th dimension in file ONLY. Therefore, need to figure out how we want to map ranks to spatial regions of the dataset, because current approach should be pretty similar to 1d array

Code now divides file into grid of mpi ranks -- does not yet work for derived types

still trying to debug derived-type -- checkpointing

Seems to be a wroking versin of exerciser.c with multidimentional arrays and chunked storage allowed

adding bool lib

changing the case of a commandline arg

changing the case of a commandline arg - correctly this time

working version of multidimensional exerciser.c

adding raw bandwidth measures to listing results

trying to be more careful about division by zero in timers

try writing timer measurements for every iteration (provide more info to a postprocessor)

changing timers to provide more info -- not finished, just a checkpoint

Four metrics are now returned with the results -- min, max, avg, std

Formatting adjustment

FIVE metrics are now returned with the results -- min, med, max, avg, std -- The minimum, median and maximum should give a good idea of the range of performance to expect

Setting NUM_ITERATIONS to 25 -- want default number of iterations to be reasonably statistically significant

saving some plotting progress - etc.

adding experimental script to plot results with layered errorbar representation

adding scripts to collect multidim data on theta

fixing a typo in subprocess call

fixing example scripts for theta

minor changes to example scripts

fixing exerplot.py for 1-3 dimensions

adding Theta results to repository -- adding png files along with raw data (may need to put this somewhere else in the future?)

moving timers -- need to experiment to determine best way to organize resuls into a meaningful way

small tweak

adding one sided run in submit.py

experimenting with custom agg perf counters -- need to test if new approach works

improving custom timer organization

changing exerciser.c to print out all custom agg performance counters for now -- this should probably be adjusted after we decide the best way to capture RMA performance

fixing bug in conversion of long to double

preparing excerciser for new batch of runs using cray-mpich-7.7.0

adding flag to turn on custom performance counters

adding the perf flag to the submit script

allowing the user to set a custom stride distance between items in local memory -- with a stride >1, the chinking is likely to be very inefficient

correcting type conversion mistake

tweaking some code -- not sure why custom perf counters are breaking the original counters

playing with the code to debug a bit

debug -- setting cutom perf counters to 1.0

try using temporary variable for reducing custom perf timers

changing interface to ccio performance counters -- returning a double now

correcting void to double error

correcting void to double error - again

new approach to organize moments

new approach to organize moments (2)

new approach to organize moments (3)

new approach to organize moments -- further re-organization

new approach to organize moments -- further re-organization (2)

fixing bug in getmoments function

further testing -- fixing const char mistake

custom stride does not work for multidimensional arrays -- custom perf timers are also breaking the code

removing real calls to ccio branch performance counters - because something needs to be DEBUGED

possible fix to this bug

new organization seems to avoid the memory bug I had before -- still need to fix local strides >1 for numdims >1

testing if same issues show up on mac and theta

fixed a few small issues -- stride==1 now works fine on theta

fixing some issues with custom stride settings and chunking of multidimensional datasets

turning custom performance timers back on

some minor cleanup -- before I forget

adding the lock timer (it will record the time for all win_lock and unlock calls)

fixing typo in new TotalWinLockTime performance timer

Multidim support and performance timer changes from Rick Zamora

Also includes some performance results.

There seems to be an issue with craympi 7.5.5 hanging (this was a workaround
for slow RMA performance at high (> 64) ppn) so just stick with current craympi
for now.

Updates to README and new submit.sh file.

Initial commit just to build it.

Initial commit to build the HDF5 custom_collective_io branch, build the
exerciser, and run the exerciser on BGQ.