GitLab

Xin Zhao authored Nov 02, 2014 and Pavan Balaji committed Nov 03, 2014

Instead of allocating / deallocating RMA operations whenever
an RMA op is posted by user, we allocate fixed size operation
pools beforehand and take the op element from those pools
when an RMA op is posted.

With only a local (per-window) op pool, the number of ops
allocated can increase arbitrarily if many windows are created.
Alternatively, if we only use a global op pool, other windows
might use up all operations thus starving the window we are
working on.

In this patch we create two pools: a local (per-window) pool and a
global pool.  Every window is guaranteed to have at least the number
of operations in the local pool.  If we run out of these operations,
we check in the global pool to see if we have any operations left.
When an operation is released, it is added back to the same pool it
was allocated from.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored Nov 02, 2014 and Pavan Balaji committed Nov 03, 2014

We were duplicating information in the operation structure and in the
packet structure when the message is actually issued.  Since most of
the information is the same anyway, this patch just embeds a packet
structure into the operation structure, so that we eliminate unnessary
copy.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored Nov 02, 2014 and Pavan Balaji committed Nov 03, 2014

The packet type MPIDI_CH3_PKT_PT_RMA_DONE is used for ACK
of FLUSH / UNLOCK packets. Here we rename it to
MPIDI_CH3_PKT_FLUSH_ACK and modify the related functions
and data structures.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored Jul 21, 2014 and Pavan Balaji committed Nov 03, 2014

We were adding an unnecessary dependency on VC structure
declarations in the mpidpkt.h file. The required information
in RMA lock queue is only the rank, but not actual VC.
Here we replace VC with rank.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored Nov 02, 2014 and Pavan Balaji committed Nov 03, 2014

Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored Nov 02, 2014 and Pavan Balaji committed Nov 03, 2014

Split RMA functionality into smaller files, and move functions
to where they belong based on the file names.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored Nov 02, 2014 and Pavan Balaji committed Nov 03, 2014

Because we are going to rewrite the RMA infrastructure
and many PVARs will no longer be used, here we temporarily
remove all PVARs and will add needed PVARs back after new
implementation is done.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Rather than having a static value for the initial size of the RTS queue,
have a CVAR to define it.
Signed-off-by: Huiwei Lu <huiweilu@mcs.anl.gov>

When the RTS queue fills up the first time, print out a warning to let
the user know that they've done it and FT won't be provided anymore.
Signed-off-by: Huiwei Lu <huiweilu@mcs.anl.gov>

req->dev.user_buf points to the data sent from origin process
to target process, and for FOP sometimes it points to the IMMED
area in packet header when data can be fit in packet header.
In such case, we should not free req->dev.user_buf in final
request handler since that data area will be freed by the
runtime when packet header is freed.

In this patch we initialize user_buf to NULL when creating the
request, and set it to NULL when FOP is completed, and avoid free
a NULL pointer in final request handler.
Signed-off-by: Min Si <msi@il.is.s.u-tokyo.ac.jp>

Signed-off-by: Devendar Bureddy <devendar@mellanox.com>
Signed-off-by: Igor Ivanov <Igor.Ivanov@itseez.com>

Xin Zhao authored Aug 12, 2014 and Pavan Balaji committed Nov 01, 2014

The original implementation includes an optimization which
allows Win_unlock for exclusive lock to return without
waiting for remote completion. This relys on the
assumption that window memory on target process will not
be accessed by a third party until that target process
finishes all RMA operations and grants the lock to other
processes. However, this assumption is not correct if user
uses assert MPI_MODE_NOCHECK. Consider the following code:

          P0                              P1           P2
    MPI_Win_lock(P1, NULL, exclusive);
    MPI_Put(X);
    MPI_Win_unlock(P1, exclusive);
    MPI_Send (P2);                                MPI_Recv(P0);
                                                  MPI_Win_lock(P1, MODE_NOCHECK, exclusive);
                                                  MPI_Get(X);
                                                  MPI_Win_unlock(P1, exclusive);

Both P0 and P2 issue exclusive lock to P1, and P2 uses assert
MPI_MODE_NOCHECK because the lock should be granted to P2 after
synchronization between P2 and P0. However, in the original
implementation, GET operation on P2 might not get the updated
value since Win_unlock on P0 return without waiting for remote
completion.

In this patch we delete this optimization. In Win_free, since every
Win_unlock guarantees the remote completion, target process no
longer needs to do additional counting works to detect target-side
completion, but only needs to do a global barrier.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

RTS messages (the first part of the LMT sequence) had no way of being
cancelled if an error occurred. This adds a small queue that keeps track
of these messages. If a failure is detected, the message is removed from
the queue and the associated request is cancelled to get out of the
progress engine.

See #1945
Signed-off-by: Huiwei Lu <huiweilu@mcs.anl.gov>

No reviewer.

There are two request handlers used when receiving data:
(1) OnDataAvail, which is triggered when data is arrived;
(2) OnFinal, which is triggered when receiving data is finished;

When receiving large derived datatype, the receiving iov can be divided
into multiple iovs. The OnDataAvail handler is set to iov load function
when still waiting for remaining data. However, such handler should be
set to OnFinal when starting receiving the last iov.

The original code does not set OnDataAvail handler to OnFinal at end.
This patch fixes this bug.

Note that this bug only appears in RMA calls, because only the RMA
packet handers need to specify OnFinal.

Resolve #2189.
Signed-off-by: Xin Zhao <xinzhao3@illinois.edu>

This patch is a workaround for an issue with older HPC-X machines.
Once we are comfortable upgrading to the latest HPC-X version, the
default value of the CVAR should be changed to true.
Signed-off-by: Xin Zhao <xinzhao3@illinois.edu>

Set reasonable limits for maximum unexpected headers and EQs at init
time. We accomplish this with a pre-init stage where we fill in a limits
struct with the system defaults, increase certain values (if they are not
set already in the environment), then do the real init.

If the "desired" limits structure had a way to allow default values for
limits we don't care about, the pre-init stage could go away.
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

Out previous increase consumed too much memory. 8MB should be enough given
the maximum put size and number of headers.
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

This reverts commit 20f1f116.

This code did not account for the fact that the first part of the message
is already sent in a PtlPut.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

The defaults used by MXM might not be fully appropriate for mpich.  So
we automatically initialize it to our preferred defaults unless the
user is trying to override it.
Signed-off-by: Xin Zhao <xinzhao3@illinois.edu>

This reverts commit 4ce4103a.

Devendar Bureddy authored Oct 25, 2014 and Pavan Balaji committed Oct 25, 2014

Do not block in mxm for blocking sends. Instead , set req pointer let
MPICH mpi layer to block on it. This will allow progress go to through
MPIDI_CH3I_Progress(), which can release the global mutex in thread
multiple case.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

The defaults used by MXM might not be fully appropriate for mpich.  So
we automatically initialize it to our preferred defaults unless the
user is trying to override it.
Signed-off-by: Xin Zhao <xinzhao3@illinois.edu>

Set the unpexected message header limit to 2 million and allocate
512MB of buffer space.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Igor Ivanov authored Oct 20, 2014 and Pavan Balaji committed Oct 23, 2014

Signed-off-by: Devendar Bureddy <devendar@mellanox.com>
Signed-off-by: Igor Ivanov <Igor.Ivanov@itseez.com>

Pavan Balaji authored Oct 21, 2014 and Kenneth Raffenetti committed Oct 22, 2014

Signed-off-by: Ken Raffenetti <raffenet@mcs.anl.gov>

In the macro MPID_nem_lmt_send_RTS, the request for the send was being freed
on an error even if it didn't exist. This caused a segfault if calling a bcast
after a process previously failed.
Signed-off-by: Huiwei Lu <huiweilu@mcs.anl.gov>

Signed-off-by: Wesley Bland <wbland@anl.gov>

Signed-off-by: Wesley Bland <wbland@anl.gov>

The calls in MPID_Comm_get_all_failed_procs and MPID_Comm_agree  were the
wrong macros for entering and exiting an MPID function. This corrects it.
Signed-off-by: Huiwei Lu <huiweilu@mcs.anl.gov>

Revoke will call MPIDI_CH3U_Clean_recvq to dequeue all requests with
revoked communicators. There is one missing case: when there's hierarchy
communicators that use a different context id. This patch adds a case to
check the hierarchy communicators.
Signed-off-by: Wesley Bland <wbland@anl.gov>

This makes sure that macros work the normal way (using a semicolon at
the end, etc.) It also removes a block of unused code from
mx_cancel.c.

Modified by Wesley to split from previous patch.
Signed-off-by: Wesley Bland <wbland@anl.gov>
Signed-off-by: Pavan Balaji <balaji@anl.gov>

We were not setting the function states correctly in a bunch of
functions.

Modifications by Wesley to split up big commit.
Signed-off-by: Wesley Bland <wbland@anl.gov>
Signed-off-by: Pavan Balaji <balaji@anl.gov>

If a message size is <= PTL_LARGE_THRESHOLD, use a single operation.
Previously, this would generate unnecessary 0-byte operations when
messages were exactly the size of the threshold.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Previously, the message pointer in an improbe call in the portals4
netmod layer was set to MPI_MESSAGE_NULL if there was no match. This
is incorrect because the ch3 layer eventually returns either a valid
pointer or NULL. As ch3 already starts the value at NULL, we can just
omit the update.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Remove incorrect event from send handler assertion. PTL_EVENT_PUT should
only be seen at the target.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Igor Ivanov authored Oct 10, 2014 and Pavan Balaji committed Oct 10, 2014

Memory leak can appear in case netmod usage. Comm override functions
are responsible for creating its own request. So they  need to come
before the sreq is created.
Signed-off-by: Igor Ivanov <Igor.Ivanov@itseez.com>
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Signed-off-by: Igor Ivanov <Igor.Ivanov@itseez.com>