GitLab

MPI_Comm_revoke is a special function because it does not have a matching call
on the "receiving side". This is because it has to act as an out-of-band,
resilient broadcast algorithm. Because of this, in this commit, in addition to
the usual functions to implement MPI communication calls (MPI/MPID/CH3/etc.),
we add a new CH3 packet type that will handle revoking a communicator without
involving a matching call from the MPI layer (similar to how RMA is currently
implemented).

The thing that must be handled most carefully when revoking a communicator is
to ensure that a previously used context ID will eventually be returned to the
pool of available context IDs and that after this occurs, no old messages will
match the new usage of the context ID (for instance, if some messages are very
slow and show up late). To accomplish this, revoke is implemented as an
all-to-all algorithm. When one process calls revoke, it will send a message to
all other processes in the communicator, which will trigger that process to
send a message to all other processes, and so on. Once a process has already
revoked its communicator locally, it won't send out another wave of messages.
As each process receives the revoke messages from the other processes, it will
track how many messages have been received. Once it has either received a
revoke message or a message about a process failure for each other process, it
will release its refcount on the communicator object. After the application
has freed all of its references to the communicator (and all requests, files,
etc. associated with it), the context ID will be returned to the available
pool.
Signed-off-by: Junchao Zhang <jczhang@mcs.anl.gov>