Bug-fix: correct the behavior of flush in exclusively locked epoch.

FLUSH should guarantee the completion of operations on both origin
and target side. However, for exclusive lock, there is an optimization
in MPICH which allows FLUSH to return without waiting for the
acknowledgement of remote completion from the target side. It relys
on the fact that there will be no other processes accessing the window
during the exclusive lock epoch.

However, such optimization is not correct when two processes allocating
windows on overlapping SHM region. Suppose P0 and P1 (on the same node)
allocate RMA window using the same SHM region, and P2 (on a different node)
locks both windows. P2 first issues a PUT and FLUSH to P0, then issues
a GET to P1 on the same memory location with PUT, since FLUSH does not
guarantee the remote completion of PUT, GET operation may not get the
updated value.

This patch disables the optimization for FLUSH and forces FLUSH to always
wait for the remote completion of operations.
Signed-off-by: Xin Zhao <xinzhao3@illinois.edu>
Signed-off-by: Antonio J. Pena <apenya@mcs.anl.gov>

src/mpid/ch3/src/ch3u_rma_sync.c

@@ -3601,11 +3601,15 @@ static int do_passive_target_rma(MPID_Win *win_ptr, int target_rank,
                  win_ptr->targets[target_rank].remote_lock_assert & MPI_MODE_NOCHECK));
 
     if (win_ptr->targets[target_rank].remote_lock_mode == MPI_LOCK_EXCLUSIVE &&
-        win_ptr->targets[target_rank].remote_lock_state != MPIDI_CH3_WIN_LOCK_CALLED) {
+        win_ptr->targets[target_rank].remote_lock_state != MPIDI_CH3_WIN_LOCK_CALLED &&
+        win_ptr->targets[target_rank].remote_lock_state != MPIDI_CH3_WIN_LOCK_FLUSH) {
         /* Exclusive lock already held -- no need to wait for rma done pkt at
            the end.  This is because the target won't grant another process
            access to the window until all of our operations complete at that
-           target.  Thus, there is no third-party communication issue. */
+           target.  Thus, there is no third-party communication issue.
+           However, flush still needs to wait for rma done, otherwise result
+           may be unknown if user reads the updated location from a shared window of
+           another target process after this flush. */
         *wait_for_rma_done_pkt = 0;
     }
     else if (MPIDI_CH3I_RMA_Ops_isempty(&win_ptr->targets[target_rank].rma_ops_list)) {