Rewrite code of piggybacking IMMED data with RMA packets.
Originally we add "immed_data" and "immed_len" areas to RMA packets, in order to piggyback small amount of data with packet header to reduce number of packets (Note that "immed_len" is necessary when the piggybacked data is not the entire data). However, those areas potentially increase the packet union size and worsen the two-sided communication. This patch fixes this issue. In this patch, we remove "immed_data" and "immed_len" from normal "MPIDI_CH3_Pkt_XXX_t" operation type (e.g. MPIDI_CH3_Pkt_put_t), and we introduce new "MPIDI_CH3_Pkt_XXX_immed_t" packt type for each operation (e.g. MPIDI_CH3_Pkt_put_immed_t). "MPIDI_CH3_Pkt_XXX_immed_t" is used when (1) both origin and target are basic datatypes, AND, (2) the data to be sent can be entirely fit into the header. By doing this, "MPIDI_CH3_Pkt_XXX_immed_t" needs "immed_data" area but can drop "immed_len" area. Also, since it only works with basic target datatype, it can drop "dataloop_size" area as well. All operations that do not satisfy (1) or (2) will use normal "MPIDI_CH3_Pkt_XXX_t" type. Originally we always piggyback FOP data into the packet header, which makes the packet size too large. In this patch we split the FOP operaton into IMMED packets and normal packets. Because CAS only work with 2 basic datatype and non-complex elements, the data amount is relatively small, we always piggyback the data with packet header and only use "MPIDI_CH3_Pkt_XXX_immed_t" packet type for CAS. Signed-off-by: Pavan Balaji <firstname.lastname@example.org>
Showing with 957 additions and 489 deletions