• Min Si's avatar
    Implement true request-based RMA operations. · 3e005f03
    Min Si authored
    There are two requests associated with each request-based
    operation: one normal internal request (req) and one newly
    added user request (ureq). We return ureq to user when
    request-based op call returns.
    The ureq is initialized with completion counter (CC) to 1
    and ref count to 2 (one is referenced by CH3 and another
    is referenced by user). If the corresponding op can be
    finished immediately in CH3, the runtime will complete ureq
    in CH3, and let user's MPI_Wait/Test to destroy ureq. If
    corresponding op cannot be finished immediately, we will
    first increment ref count to 3 (because now there are
    three places needed to reference ureq: user, CH3,
    progress engine). Progress engine will complete ureq when
    op is completed, then CH3 will release its reference during
    garbage collection, finally user's MPI_Wait/Test will
    destroy ureq.
    The ureq can be completed in following three ways:
    1. If op is issued and completed immediately in CH3
    (req is NULL), we just complete ureq before free op.
    2. If op is issued but not completed, we remember the ureq
    handler in req and specify OnDataAvail / OnFinal handlers
    in req to a newly added request handler, which will complete
    user reqeust. The handler is triggered at three places:
       2-a. when progress engine completes a put/acc req;
       2-b. when get/getacc handler completes a get/getacc req;
       2-c. when progress engine completes a get/getacc req;
    3. If op is not issued (i.e., wait for lock granted), the 2nd
    way will be eventually performed when such op is issued by
    progress engine.
    Signed-off-by: default avatarXin Zhao <xinzhao3@illinois.edu>