• Xin Zhao's avatar
    Rewrite code of piggybacking IMMED data with RMA packets. · de9d0f21
    Xin Zhao authored
    Originally we add "immed_data" and "immed_len" areas to RMA packets,
    in order to piggyback small amount of data with packet header to
    reduce number of packets (Note that "immed_len" is necessary when
    the piggybacked data is not the entire data). However, those areas
    potentially increase the packet union size and worsen the two-sided
    communication. This patch fixes this issue.
    In this patch, we remove "immed_data" and "immed_len" from normal
    "MPIDI_CH3_Pkt_XXX_t" operation type (e.g. MPIDI_CH3_Pkt_put_t), and
    we introduce new "MPIDI_CH3_Pkt_XXX_immed_t" packt type for each
    operation (e.g. MPIDI_CH3_Pkt_put_immed_t).
    "MPIDI_CH3_Pkt_XXX_immed_t" is used when (1) both origin and target
    are basic datatypes, AND, (2) the data to be sent can be entirely fit
    into the header. By doing this, "MPIDI_CH3_Pkt_XXX_immed_t" needs
    "immed_data" area but can drop "immed_len" area. Also, since it only
    works with basic target datatype, it can drop "dataloop_size" area
    as well. All operations that do not satisfy (1) or (2) will use
    normal "MPIDI_CH3_Pkt_XXX_t" type.
    Originally we always piggyback FOP data into the packet header,
    which makes the packet size too large. In this patch we split the
    FOP operaton into IMMED packets and normal packets.
    Because CAS only work with 2 basic datatype and non-complex
    elements, the data amount is relatively small, we always piggyback
    the data with packet header and only use "MPIDI_CH3_Pkt_XXX_immed_t"
    packet type for CAS.
    Signed-off-by: Pavan Balaji's avatarPavan Balaji <balaji@anl.gov>