Darshan already has the ability to use mmap() to manage memory for it's runtime statistics collection, but it also uses a few small malloc() calls at runtime for transient data structures. This can cause problems when interacting with other instrumentation libraries that could invoke POSIX file operations in the malloc path and cause a deadlock (examples include libunwind, hugepages, and alternative malloc implementations).

We've been able to work around this case by case in the past, but John Mellor-Crummey offered this suggestion for a more systemic solution:

What we do in HPCToolkit is just mmap a segment of perhaps 4MB. Then our allocator simply advances a cursor through the segment to provide data. There is no free. I assume that you don’t need to free either. Then, if we out of space in the segment, we mmap another and move the cursor there. Our code as it is a bit specific to HPCToolkit, so you probably don’t want it verbatim. However, it would probably provide a good template for what to do. You can find the few files involved here:

https://github.com/HPCToolkit/hpctoolkit/tree/master/src/tool/hpcrun/memory