This seems to have been going on for a long time, but the "pipelined" bake test segfaults at shutdown on cooley.

rm -f /dev/shm/foo.dat
bake-mkpool -s 60G /dev/shm/foo.dat
mpirun -f $COBALT_NODEFILE -n 2 numactl -N 1 -m 1 catchsegv ./bake-p2p-bw -x 16777216 -m 34359738368 -n "verbs://" -p /dev/shm/foo.dat -c 1 -r 4 -i 

The earlier tests (without the '-i') do not segfault.

The backtrace looks like this:

(gdb) bt
#0  0x00007ff00b0f85e2 in margo_bulk_pool_destroy (pool=0x4) at src/margo-bulk-pool.c:127
#1  0x00007ff00b0f89fd in margo_bulk_poolset_destroy (poolset=0x473d620) at src/margo-bulk-pool.c:290
#2  0x00007ff00c656ca7 in bake_server_finalize_cb (data=0x237ae30) at src/bake-server.c:1840
#3  0x00007ff00b0f303a in margo_cleanup (mid=0x224f7d0) at src/margo.c:549
#4  0x00007ff00b0f70bc in margo_finalize (mid=<optimized out>) at src/margo.c:669
#5  0x0000000000402ebb in main (argc=<optimized out>, argv=<optimized out>) at ../perf-regression/bake-p2p-bw.c:275

The bake server finalize callback is trying to clean up a memory pool:

  1839     if(provider->poolset)
  1840         margo_bulk_poolset_destroy(provider->poolset);