Here we wrap up common action when one RMA op is finished on target
into a function to make code structure cleaner.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

When operation pending list and request lists are all empty, FLUSH message
needs to be sent by origin only when origin issued PUT/ACC operations since
the last synchronization calls, otherwise origin does not need to issue FLUSH
at all and does not need to wait for FLUSH ACK message.

Similiarly, origin waits for ACK of UNLOCK message only when origin issued
PUT/ACC operations since the last synchronization calls. However, UNLOCK
message always needs to be sent out because origin needs to unlock the
target process. This patch avoids issuing unnecessary
FLUSH / FLUSH ACK / UNLOCK ACK messages.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

There are two request handlers used when receiving data:
(1) OnDataAvail, which is triggered when data is arrived;
(2) OnFinal, which is triggered when receiving data is finished;

In progress engine, only OnDataAvail is triggered when a request is
completed. The upper ch3 layer should change OnDataAvail to OnFinal when
the coming receiving data will complete the request.

However, in the original implementation, when receiving multiple
segments for a large receive data, the OnDataAvail was reset to 0
at the last segment hence the final action was lost. This patch fixed
this bug.

In RMA target put/acc/gacc packet handlers, OnDataAvail was reset to
OnFinal function if OnDataAvail is 0 due to this bug. This patch also
rewrites this part so that packet handlers only sets proper OnFinal
handler at beginning and let the receiving data function change
OnDataAvail to OnFinal at the last segment.

Signed-off-by: Xin Zhao <xinzhao3@illinois.edu>

There are two requests associated with each request-based
operation: one normal internal request (req) and one newly
added user request (ureq). We return ureq to user when
request-based op call returns.

The ureq is initialized with completion counter (CC) to 1
and ref count to 2 (one is referenced by CH3 and another
is referenced by user). If the corresponding op can be
finished immediately in CH3, the runtime will complete ureq
in CH3, and let user's MPI_Wait/Test to destroy ureq. If
corresponding op cannot be finished immediately, we will
first increment ref count to 3 (because now there are
three places needed to reference ureq: user, CH3,
progress engine). Progress engine will complete ureq when
op is completed, then CH3 will release its reference during
garbage collection, finally user's MPI_Wait/Test will
destroy ureq.

The ureq can be completed in following three ways:

1. If op is issued and completed immediately in CH3
(req is NULL), we just complete ureq before free op.

2. If op is issued but not completed, we remember the ureq
handler in req and specify OnDataAvail / OnFinal handlers
in req to a newly added request handler, which will complete
user reqeust. The handler is triggered at three places:
   2-a. when progress engine completes a put/acc req;
   2-b. when get/getacc handler completes a get/getacc req;
   2-c. when progress engine completes a get/getacc req;

3. If op is not issued (i.e., wait for lock granted), the 2nd
way will be eventually performed when such op is issued by
progress engine.

Signed-off-by: Xin Zhao <xinzhao3@illinois.edu>

Add some original RMA PVARs back to the new
RMA infrastructure, including timing of packet
handlers, op allocation and setting, window
creation, etc.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

We made a huge change to RMA infrastructure and
a lot of old code can be droped, including separate
handlers for lock-op-unlock, ACCUM_IMMED specific
code, O(p) data structure code, code of lazy issuing,
etc.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

1. Piggyback LOCK request with first IMMED operation.

When we see an IMMED operation, we can always piggyback
LOCK request with that operation to reduce one sync
message of single LOCK request. When packet header of
that operation is received on target, we will try to
acquire the lock and perform that operation. The target
either piggybacks LOCK_GRANTED message with the response
packet (if available), or sends a single LOCK_GRANTED
message back to origin.

2. Rewrite code of manage lock queue.

When the lock request cannot be satisfied on target,
we need to buffer that lock request on target. All we
need to do is enqueuing the packet header, which contains
all information we need after lock is granted. When
the current lock is released, the runtime will goes
over the lock queue and grant the lock to the next
available request. After lock is granted, the runtime
just trigger the packet handler for the second time.

3. Release lock on target side if piggybacking with UNLOCK.

If there are active-message operations to be issued,
we piggyback a UNLOCK flag with the last operation.
When the target recieves it, it will release the current
lock and grant the lock to the next process.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

For FOP operation, all data can be fit into the packet
header, so on origin side we do not need to send separate
data packets, and on target side we do not need request
handler, only packet handler is needed. Similar with FOP
response packet, we can receive all data in FOP resp packet
handler. This patch delete the request handler on target
side and simplify packet handler on target / origin side.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

We add a IMMED data area (16 bytes by default) in
packet header which will contains as much origin
data as possible. If origin can put all data in
packet header, then it no longer needs to send
separate data packet. When target recieves the
packet header, it will first copy data out from
the IMMED data area. If there is still more
data coming, it continues to receive following
packets; if all data is included in header, then
recieving is done.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

During PSCW, when there are active-message operations
to be issued in Win_complete, we piggback a AT_COMPLETE
flag with it so that when target receives it, it can
decrement a counter on target side and detect completion
when target counter reaches zero.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

When the origin wants to do a FLUSH sync, if there are
active-message operations that are going to be issued,
we piggback the FLUSH message with the last operation;
if no such operations, we just send a single FLUSH packet.

If the last operation is a write op (PUT, ACC) or only
a single FLUSH packet is sent, after target recieves it,
target will send back a single FLUSH_ACK packet;
if the last operation contains a read action (GET, GACC, FOP,
CAS), after target receiveds it, target will piggback a
FLUSH_ACK flag with the response packet.

After origin receives the FLUSH_ACK packet or response packet
with FLUSH_ACK flag, it will decrement the counter which
indicates number of outgoing sync messages (FLUSH / UNLOCK).
When that counter reaches zero, origin can know that remote
completion is achieved.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

Separate final request handler of PUT, ACC, GACC into three.
Separate derived DT request handler of ACC and GACC into two.

Renaming request handlers as follows:

(1) Normal request handler: it is triggered on target side
    when all data from origin is received.

    It includes:

    ReqHandler_PutRecvComplete --- for PUT
    ReqHandler_AccumRecvComplete --- for ACC
    ReqHandler_GaccumRecvComplete --- for GACC

(2) Derived DT request handler: it is triggered on target
    side when all derived DT info is recieved.

    It includes:

    ReqHandler_PutDerivedDTRecvComplete --- for PUT
    ReqHandler_AccumDerivedDTRecvComplete --- for ACC
    ReqHandler_GaccumDerivedDTRecvComplete --- for GACC

(3) Reponse request handler: it is triggered on target
    side when sending back process is finished in GET-like
    operations.

    It includes:

    ReqHandler_GetSendComplete --- for GET
    ReqHandler_GaccumLikeSendComplete --- for GACC, FOP, CAS

Signed-off-by: Pavan Balaji <balaji@anl.gov>

Previously several RMA packet types share the same structure,
which is misleading for coding. Here make different
RMA packet types use different packet data structures.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored Nov 02, 2014 and Pavan Balaji committed Nov 03, 2014

We were duplicating information in the operation structure and in the
packet structure when the message is actually issued.  Since most of
the information is the same anyway, this patch just embeds a packet
structure into the operation structure, so that we eliminate unnessary
copy.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored Nov 02, 2014 and Pavan Balaji committed Nov 03, 2014

The packet type MPIDI_CH3_PKT_PT_RMA_DONE is used for ACK
of FLUSH / UNLOCK packets. Here we rename it to
MPIDI_CH3_PKT_FLUSH_ACK and modify the related functions
and data structures.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored Jul 21, 2014 and Pavan Balaji committed Nov 03, 2014

We were adding an unnecessary dependency on VC structure
declarations in the mpidpkt.h file. The required information
in RMA lock queue is only the rank, but not actual VC.
Here we replace VC with rank.

Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored Nov 02, 2014 and Pavan Balaji committed Nov 03, 2014

Split RMA functionality into smaller files, and move functions
to where they belong based on the file names.

Signed-off-by: Pavan Balaji <balaji@anl.gov>