GitLab

Xin Zhao authored Apr 30, 2015 and Pavan Balaji committed May 30, 2015

This reverts commit 19f29078

.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored Apr 29, 2015 and Pavan Balaji committed May 30, 2015

Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored May 18, 2015 and Pavan Balaji committed May 30, 2015

Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored May 18, 2015 and Pavan Balaji committed May 30, 2015

In RMA communication, when data to be sent is derived datatype but
contiguous, we can avoid copy/packing but directly issue it from
the user buffer. However, the data may have non-zero lb and we should
add the true lb when calculating the starting address. This patch
fixes this issue.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored May 28, 2015 and Pavan Balaji committed May 30, 2015

Originally the implementation of sendNoncontig() in Nemesis
assume that req->dev.segment_first is 0 and req->dev.segment_size
is the size of data, which potentially asssumes that the data passed
from upper layer must be the entire data, prohibiting the possibility
of streaming that data in the upper layer. The general way to use them
should be that req->dev.segment_first specifies the current starting
location and req->dev.segment_size specifies the current ending location.

We fixed this issue in commit 5132e070

, but
there are some places missed in that fixing. Here we fixed those places.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Xin Zhao authored May 28, 2015 and Pavan Balaji committed May 30, 2015

Note that here we move the definition of streaming unit size to
mpidimpl.h in CH3, so that Nemesis can see it and do assert check on it.
Signed-off-by: Pavan Balaji <balaji@anl.gov>

Two tag bits are reserved for error propagation. We need to make sure
they are ignored by the network matching capabilities. Refs #2260

No reviewer.

Halim Amer authored May 26, 2015 and Kenneth Raffenetti committed May 27, 2015

If the destination of a LMT nonblocking send, which
follows a rendez-vous protocol, receives the RTS packet to the
unexpected queue followed by a cancelation request, the request in
the unexpected queue never gets freed. To solve the issue, we
forcefully free the rendez-vous request by adding an additional
request release operation. Refs #287
Signed-off-by: Ken Raffenetti <raffenet@mcs.anl.gov>

Frees memory allocated to track large sends when those sends are
successfully cancelled.

No reviewer.

Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

Simplifies request completion logic and ensures operations are properly
accounted for in the cancel send case. Removes pt2pt send tracking in the
VC, as requests are enough to guarantee completion.
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

Pavan Balaji authored May 13, 2015 and Kenneth Raffenetti committed May 13, 2015

Signed-off-by: Ken Raffenetti <raffenet@mcs.anl.gov>

Pavan Balaji authored May 12, 2015 and Kenneth Raffenetti committed May 13, 2015

Signed-off-by: Ken Raffenetti <raffenet@mcs.anl.gov>

Signed-off-by: Jithin Jose <jithin.jose@intel.com>
Signed-off-by: Charles J Archer <charles.j.archer@intel.com>

Signed-off-by: Jithin Jose <jithin.jose@intel.com>
Signed-off-by: Charles J Archer <charles.j.archer@intel.com>

This includes several changes:

1. Merged WRAPPER and EXTERNAL LIBS.  There is no reason to maintain
two names for these flags.  These are eventually appended and added to
the compiler wrappers anyway.

2. Updated mpicc and friend to add the necessary flags directly
instead of trying to merge these flags in configure.
Signed-off-by: Ken Raffenetti <raffenet@mcs.anl.gov>

MPID_nem_ofi_data_callback used to check sreq->cc in order to track progress of
the RTS/CTS/DATA protocol. The was an implicit assumption that fi_tsend with RTS
completes first. However this would cause a hang if fi_trecv completed earlier.
The fix is: don't rely on the cc but rather check the tag bits explicitly.
Note, the RTS/CTS/DATA bits are no longer accumulated (i.e., no more
"wc->tag | CTS/DATA").
Signed-off-by: Charles J Archer <charles.j.archer@intel.com>

It uses nemesis shared memory which is already cleaned up at this stage.
However, w/o any synchronization a hang in the close protocol is possible
since rts/cts/data messages may be on the fly. This change fixes the issue.
Signed-off-by: Charles J Archer <charles.j.archer@intel.com>

Signed-off-by: Charles J Archer <charles.j.archer@intel.com>

Signed-off-by: Charles J Archer <charles.j.archer@intel.com>

This patch modifies the OFI netmod to support large tag layouts, while preserving the old
tag layout.  OFI defines a 64 bit tag, but also provides for a 64 bit tag and immediate data.
In some OFI providers, we may want to select different tag layouts.  This patch currently
does not query for the proper tag layout or attempt to make a choice of the optimal layout,
it provides macro/templatized support for different tag formats.  Additional selection
criteria will be added in subsequent patches.

  * Tag layout is moved to a separate file.
    Added init_sendtag_M2, init_recvtag_M2 (M2 stands for MODE #2, i.e. the mode
    that uses fi_tsenddata and does not pack source into tag).

  * Created a template file for ofi_tagged.c
    Moved do_isend into template file which is included twice into ofi_tagged.c thus providing for the two
    versions of do_isend and do_isend_2 corresponding to the two API sets.

  * All send functions are available in two versions.
    Added macro that declares a function for the two API sets. The first set has the namings inherited from
    the previous netmod version. The functions of the second API set have the "_2" suffix.

  * Recv_posted, anysource_posted, recv_callback, ofi_probe  are templatized.

  * ofi_tag_to_vc renamed ofi_wc_to_vc
    Note, for the API_SET_2 the pgid is stored in the imm data while
    psource and port will be packed the same way as in API_SET_1.

  * Adds api_set member in gl_data struct.  Initialize routines based on api_set

  * Added RCD (RtsCtsData) protocol identifiers

  * Added support for OFI MEM_TAG_FORMAT

  * PGID placement modified
Signed-off-by: Charles J Archer <charles.j.archer@intel.com>

Igor Ivanov authored Apr 21, 2015 and Pavan Balaji committed Apr 24, 2015

Signed-off-by: Igor Ivanov <Igor.Ivanov@itseez.com>

Instead of polling for an arbitrarily decided number of times in the
progress engine before yielding, we now moved the yielding
intelligence to the threading layer.  The threading layer can keep
track of other threads that are waiting to enter the critical section
and only yield if another thread is waiting.  In this way, if no
thread is waiting to get the lock, the main thread never yields.  At
the same time, if another thread is waiting to get a lock, there is no
delay in yielding.

This change, however, introduces possible deadlocks. If a thread enters
MPIDI_CH3I_progress with is_blocking unset, it may set the
MPIDI_CH3I_progress_blocked flag and then will yield the critical section.
Another thread may enter with is_blocking set, find the flag
MPIDI_CH3I_progress_blocked set, and block in the conditional variable.
The first thread will wake up and leave the progress engine without
emitting any signal to wake up the second thread which may sleep forever.

A simple fix is to yield the critical section only if the current thread
entered the progress engine with is_blocking set.
Signed-off-by: Halim Amer <aamer@anl.gov>

The nemesis progress engine was written in a way so that if one thread
is inside a progress engine, other threads cannot enter the receive
progress.  They can enter the send progress in some cases.  There
doesn't seem to be a good reason for this behavior.  This patch
combines this so threads would simply return for nonblocking
operations and wait for a signal before entering the progress engine
for blocking operations.
Signed-off-by: Halim Amer <aamer@anl.gov>

The return value of anysource_matched should be the actual result
of the cancel operation. If the result is uncancelable, i.e. already
matched, then CH3 will let the netmod message win and move on to the
other requests in the queue. When the completion for the unsuccessfully
canceled message comes in, we process it like normal.
Reviewed-by: Igor Ivanov <Igor.Ivanov@itseez.com>
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

Signed-off-by: Min Si <msi@il.is.s.u-tokyo.ac.jp>
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

Here we should check if the packet type is FOP_IMMED, if so,
we initialize the response packet to be FOP_RESP_IMMED. The
originally code wrongly check the packet flag instead of packet
type.
Signed-off-by: Min Si <msi@il.is.s.u-tokyo.ac.jp>
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

After reducing the IMMED data size from 16 bytes to 8 bytes,
FOP data is no longer always fit in the packet header, hence
the assert no longer makes sense.
Signed-off-by: Min Si <msi@il.is.s.u-tokyo.ac.jp>
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

Originally the size of IMMED data in RMA packets is 16 bytes
which makes the size of CH3 packet be 56 bytes. Here we reduce
the size of IMMED data in RMA packets to 8 bytes, so that the
size of CH3 packet is reduced to 48 bytes, the same with
mpich-3.1.4 (the old RMA infrastructure).
Signed-off-by: Min Si <msi@il.is.s.u-tokyo.ac.jp>
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

'stream_offset' is used to specify the starting position
(on target window) of the current streaming unit in ACC-like
operations. It is originally put in the RMA packet struct,
which potentially increases the size of CH3 packet size.

In this patch, we move 'stream_offset' out of the RMA
packet as follows: 1. when target data is basic datatype,
we use 'stream_offset' and the starting address for the entire
operation to calculate the starting address for current
streaming unit, and rewrite 'addr' in RMA packet with that
value; 2. when target data is derived datatype, we cannot do
the same thing as basic datatype because the target needs to
know both the starting address for the entire operation and
the starting address for the current streaming unit. Therefore,
we send 'stream_offset' separately to the target side.
Signed-off-by: Min Si <msi@il.is.s.u-tokyo.ac.jp>
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

An assert protecting from a non-null request was happening too late in pkt_COOKIE_handler from
mpid_nem_lmt.c. This patch moves it to an earlier location so that it's checked before it's first
used.

Reported by Dmitry Polyakov.

Update OFI netmod to match portals4 netmod anysource_matched semantics.

This reverts commit 17e31e59.

This fix, along with a pending patch to the Portal4 reference implementation,
should make anysource_matched a more reliable operation for multithreaded apps. We
were seeing a race condition where an ME would unlink successfully, but an
event matching it would still arrive in the queue. CH3 can now reliably search
the netmod queue for matched MPI_ANY_SOURCE requests.

The reason that we no longer assert that an MPI_ANY_SOURCE request was removed
from the CH3 queue is that FDP (find and dequeue posted) operations will remove
the request from the queue, if it is known to be already matched by the netmod,
even if it has not yet completed.

Fixes #2199
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

 * Rename MPIR_CVAR_DUMP_PROVIDERS to MPIR_CVAR_OFI_DUMP_PROVIDERS
 * Add MPIR_CVAR_OFI_USE_PROVIDER, which takes a string to desired
   provider name

Changes the value of various static limits in the Portals4 netmod, based
on experimentation results and suggestions from collaborators.

1. Bump most ni_limits from 32K to 64K. These limits relate closely to
   queue depth. We can reasonably expect to support a queue depth
   of 64K.

2. Limit issued origin events to 500. This translates to sending ~250
   operations to Portals at a time, which over IB is roughly the
   saturation point. TODO: turn this into a CVAR.

3. Limit per target issued operations to 50. This will give the target a
   better chance to process events without being overwhelmed by a single
   process. TODO: turn this into a CVAR, also.

4. Allocate more buffer space for incoming control messages. Observed
   results, especially with larger messages, showed that more buffer space
   cuts down on flow-control events.
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>