Add global / local pools of RMA ops and related APIs.
Instead of allocating / deallocating RMA operations whenever
an RMA op is posted by user, we allocate fixed size operation
pools beforehand and take the op element from those pools
when an RMA op is posted.
With only a local (per-window) op pool, the number of ops
allocated can increase arbitrarily if many windows are created.
Alternatively, if we only use a global op pool, other windows
might use up all operations thus starving the window we are
working on.
In this patch we create two pools: a local (per-window) pool and a
global pool. Every window is guaranteed to have at least the number
of operations in the local pool. If we run out of these operations,
we check in the global pool to see if we have any operations left.
When an operation is released, it is added back to the same pool it
was allocated from.
Signed-off-by:
Pavan Balaji <balaji@anl.gov>