Preparing beta version for new dragonfly model, modifications to routings,...

Preparing beta version for new dragonfly model, modifications to routings, adding documentation for generating config files, adding libtools to support C++ code compilation

CONTRIBUTORS.md

-Nikhil Jain (UIUC)
+Nikhil Jain, Abhinav Bhatele (LLNL)
     - Improvements in credit-based flow control of CODES dragonfly and torus network models.
     - Addition of direct scheme for setting up dragonfly network topology.
+    - Network configuration setup for custom dragonfly model.
+    - Topology generations scripts for custom dragonfly model.
     - Bug reporter for CODES network models.
+    - Fat tree network setup and adaptive routing.
+
+Jens Domke (U. of Dresden)
+    - Static routing in fat tree network model.
 
 Xu Yang (IIT)
     - Added support for running multiple application workloads with CODES MPI

doc/GETTING_STARTED

@@ -319,7 +319,7 @@ transparent message passing support between LPs, allowing the underlying
 network LPs to do the work of routing while user LPs model their applications.
 It consists of a number of both simple and complex network models as well as
 configuration utilities and communication APIs. A somewhat stale overview is
-also given at src/models/networks/model-net/doc/README.
+also given at src/networks/model-net/doc/README.
 
 = Components of model-net
 
@@ -407,8 +407,8 @@ for the latency and bandwidth costs: "net_latency_ns_file" and
 "net_bw_mbps_file".
 
 More details about the models can be found at
-src/models/networks/model-net/doc/README.simplenet.txt and
-src/models/networks/model-net/doc/README.simplep2p.txt, respectively.
+src/networks/model-net/doc/README.simplenet.txt and
+src/networks/model-net/doc/README.simplep2p.txt, respectively.
 
 == LogGP
 
@@ -422,7 +422,7 @@ The only configuration entry the LogGP model requires is
 configuration file.
 
 For more details on gathering parameters for the LogGP model, as well as it's
-usage and caveats, see the document src/models/model-net/doc/README.loggp.txt. 
+usage and caveats, see the document src/model-net/doc/README.loggp.txt. 
 
 == Torus
 
@@ -435,7 +435,7 @@ performed in:
     (PADS), 2014.
 
 The configuration and model setup can be found at:
-src/models/model-net/doc/README.torus.txt
+src/model-net/doc/README.torus.txt
 
 == Dragonfly
 
@@ -443,7 +443,7 @@ The dragonfly model (model-net LP name: "dragonfly") is a network
 topology that utilizes the concept of virtual routers to produce systems with
 very high virtual radix out of network components with a lower radix. The
 topology itself and the simulation model are both described in
-src/models/networks/model-net/doc/README.dragonfly.txt.
+src/networks/model-net/doc/README.dragonfly.txt.
 cite).
 
 The configuration parameters are a little trickier here, as additional LPs
@@ -453,7 +453,7 @@ represents a physical router. At least one "dragonfly_router" LP must be
 present in every LP group with a "modelnet_dragonfly" LP. 
 
 Further configuration and model setup can be found at
-src/models/model-net/doc/README.dragonfly.txt.
+src/model-net/doc/README.dragonfly-custom.txt.
 
 = CODES example model
 

scripts/gen-cray-topo/README.txt

+** Generating inter and intra group files for Edison and Theta Interconnects **:
+
+- Edison network config files:
+    python gen-network-config.py link-edison.txt intra-edison inter-edison
+
+- Theta network config files:
+    python gen-network-config.py theta intra-theta inter-theta
+
+ ** Generating customizable dragonfly interconnects **:
+
+mpicc connections_general.c -o connections_general
+
+./connections_general g r c intra-file inter-file
+
+--> g: number of groups in the network
+--> r: number of router rows within a group
+--> c: number of router columns within a group
+--> intra-file: output files for intra-group connections
+--> inter-file: output file for inter-group connections
+
+- The scripts and code for translating existing topologies and generating
+  cray-style dragonfly topologies have been contributed by Nikhil Jain, Abhinav
+  Bhatele and Peer-Timo Breemer from LLNL.
+
+- For details on cray XC dragonfly network topology, see the following paper:
+
+@inproceedings{faanes2012cray,
+    title={Cray cascade: a scalable HPC system based on a Dragonfly network},
+    author={Faanes, Greg and Bataineh, Abdulla and Roweth, Duncan and Froese,
+    Edwin and Alverson, Bob and Johnson, Tim and Kopnick, Joe and Higgins, Mike
+    and Reinhard, James and others},
+    booktitle={Proceedings of the International Conference on High
+    Performance Computing, Networking, Storage and Analysis},
+    pages={103},
+    year={2012},
+    organization={IEEE Computer Society Press}
+}

scripts/connections_general.c → scripts/gen-cray-topo/connections_general.c

@@ -21,16 +21,16 @@
 
 int main(int argc, char **argv) {
   if(argc < 3) {
-    printf("Corret usage: %s <num_g> <intra_file> <inter_file>", argv[0]);
+    printf("Correct usage: %s <num_g> <num_rows> <num_cols> <intra_file> <inter_file>", argv[0]);
     exit(0);
   }
 
   int g = atoi(argv[1]);
-  int r = 6;
-  int c = 16;
+  int r = atoi(argv[2]);
+  int c = atoi(argv[3]);
 
-  FILE *intra = fopen(argv[2], "wb");
-  FILE *inter = fopen(argv[3], "wb");
+  FILE *intra = fopen(argv[4], "wb");
+  FILE *inter = fopen(argv[5], "wb");
 
   int router = 0;
   int green = 0, black = 1;

src/Makefile.subdir

@@ -175,6 +175,7 @@ bin_PROGRAMS += src/workload/codes-workload-dump
 bin_PROGRAMS += src/network-workloads/model-net-mpi-replay
 bin_PROGRAMS += src/network-workloads/model-net-dumpi-traces-dump
 bin_PROGRAMS += src/network-workloads/model-net-synthetic
+bin_PROGRAMS += src/network-workloads/model-net-synthetic-custom-dfly
 bin_PROGRAMS += src/network-workloads/model-net-synthetic-slimfly
 
 src_workload_codes_workload_dump_SOURCES = \
@@ -182,6 +183,7 @@ src_workload_codes_workload_dump_SOURCES = \
 
 src_network_workloads_model_net_mpi_replay_SOURCES = src/network-workloads/model-net-mpi-replay.c 
 src_network_workloads_model_net_synthetic_SOURCES = src/network-workloads/model-net-synthetic.c 
+src_network_workloads_model_net_synthetic_custom_dfly_SOURCES = src/network-workloads/model-net-synthetic-custom-dfly.c 
 src_network_workloads_model_net_synthetic_slimfly_SOURCES = src/network-workloads/model-net-synthetic-slimfly.c src/network-workloads/dummy.C 
 src_network_workloads_model_net_dumpi_traces_dump_SOURCES = src/network-workloads/model-net-dumpi-traces-dump.c src/network-workloads/dummy.C
 

src/network-workloads/conf/modelnet-synthetic-edison.conf

+LPGROUPS
+{
+   MODELNET_GRP
+   {
+      repetitions="1440";
+# name of this lp changes according to the model
+      server="4";
+# these lp names will be the same for dragonfly-custom model
+      modelnet_dragonfly_custom="4";
+      modelnet_dragonfly_custom_router="1";
+   }
+}
+PARAMS
+{
+# packet size in the network 
+   packet_size="1024";
+   modelnet_order=( "dragonfly_custom","dragonfly_custom_router" );
+   # scheduler options
+   modelnet_scheduler="fcfs";
+# chunk size in the network (when chunk size = packet size, packets will not be
+# divided into chunks)
+   chunk_size="1024";
+   # modelnet_scheduler="round-robin";
+   # number of routers within each group
+   # this is dictated by the dragonfly configuration files
+   num_routers="96";
+   # number of groups in the network
+   num_groups="15";
+# buffer size in bytes for local virtual channels 
+   local_vc_size="8192";
+#buffer size in bytes for global virtual channels 
+   global_vc_size="16384";
+#buffer size in bytes for compute node virtual channels 
+   cn_vc_size="8192";
+#bandwidth in GiB/s for local channels 
+   local_bandwidth="5.25";
+# bandwidth in GiB/s for global channels 
+   global_bandwidth="4.7";
+# bandwidth in GiB/s for compute node-router channels 
+   cn_bandwidth="8.0";
+# ROSS message size 
+   message_size="584";
+# number of compute nodes connected to router, dictated by dragonfly config
+# file
+   num_cns_per_router="4";
+# number of global channels per router 
+   num_global_channels="8";
+# network config file for intra-group connections 
+   intra-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/intra-edison";
+# network config file for inter-group connections
+   inter-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/inter-edison";
+# routing protocol to be used 
+   routing="adaptive";
+}

src/network-workloads/conf/modelnet-test-dragonfly-custom.conf

@@ -2,32 +2,53 @@ LPGROUPS
 {
    MODELNET_GRP
    {
-      repetitions="1440";
-      nw-lp="1";
-      modelnet_dragonfly_custom="1";
+      repetitions="3200";
+# name of this lp changes according to the model
+      nw-lp="4";
+# these lp names will be the same for dragonfly-custom model
+      modelnet_dragonfly_custom="4";
       modelnet_dragonfly_custom_router="1";
    }
 }
 PARAMS
 {
-   packet_size="512";
+# packet size in the network 
+   packet_size="1024";
    modelnet_order=( "dragonfly_custom","dragonfly_custom_router" );
    # scheduler options
    modelnet_scheduler="fcfs";
-   chunk_size="256";
+# chunk size in the network (when chunk size = packet size, packets will not be
+# divided into chunks)
+   chunk_size="1024";
    # modelnet_scheduler="round-robin";
-   num_routers="96";
-   num_groups="15";
-   local_vc_size="2048";
-   global_vc_size="8192";
-   cn_vc_size="1024";
+   # number of routers within each group
+   # this is dictated by the dragonfly configuration files
+   num_routers="128";
+   # number of groups in the network
+   num_groups="25";
+# buffer size in bytes for local virtual channels 
+   local_vc_size="8192";
+#buffer size in bytes for global virtual channels 
+   global_vc_size="16384";
+#buffer size in bytes for compute node virtual channels 
+   cn_vc_size="8192";
+#bandwidth in GiB/s for local channels 
    local_bandwidth="5.25";
+# bandwidth in GiB/s for global channels 
    global_bandwidth="4.7";
-   cn_bandwidth="5.25";
+# bandwidth in GiB/s for compute node-router channels 
+   cn_bandwidth="8.0";
+# ROSS message size 
    message_size="584";
-   num_cns_per_router="1";
-   num_global_channels="4";
-   intra-group-connections="intracray2";
-   inter-group-connections="intercray2";
+# number of compute nodes connected to router, dictated by dragonfly config
+# file
+   num_cns_per_router="4";
+# number of global channels per router 
+   num_global_channels="8";
+# network config file for intra-group connections 
+   intra-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/intra-custom";
+# network config file for inter-group connections
+   inter-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/inter-custom";
+# routing protocol to be used 
    routing="prog-adaptive";
 }

src/network-workloads/conf/modelnet-test-dragonfly-edison.conf

+LPGROUPS
+{
+   MODELNET_GRP
+   {
+      repetitions="1440";
+# name of this lp changes according to the model
+      nw-lp="4";
+# these lp names will be the same for dragonfly-custom model
+      modelnet_dragonfly_custom="4";
+      modelnet_dragonfly_custom_router="1";
+   }
+}
+PARAMS
+{
+# packet size in the network 
+   packet_size="1024";
+   modelnet_order=( "dragonfly_custom","dragonfly_custom_router" );
+   # scheduler options
+   modelnet_scheduler="fcfs";
+# chunk size in the network (when chunk size = packet size, packets will not be
+# divided into chunks)
+   chunk_size="1024";
+   # modelnet_scheduler="round-robin";
+   # number of routers within each group
+   # this is dictated by the dragonfly configuration files
+   num_routers="96";
+   # number of groups in the network
+   num_groups="15";
+# buffer size in bytes for local virtual channels 
+   local_vc_size="8192";
+#buffer size in bytes for global virtual channels 
+   global_vc_size="16384";
+#buffer size in bytes for compute node virtual channels 
+   cn_vc_size="8192";
+#bandwidth in GiB/s for local channels 
+   local_bandwidth="5.25";
+# bandwidth in GiB/s for global channels 
+   global_bandwidth="4.7";
+# bandwidth in GiB/s for compute node-router channels 
+   cn_bandwidth="8.0";
+# ROSS message size 
+   message_size="584";
+# number of compute nodes connected to router, dictated by dragonfly config
+# file
+   num_cns_per_router="4";
+# number of global channels per router 
+   num_global_channels="8";
+# network config file for intra-group connections 
+   intra-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/intra-edison";
+# network config file for inter-group connections
+   inter-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/inter-edison";
+# routing protocol to be used 
+   routing="adaptive";
+}

src/network-workloads/conf/modelnet-test-dragonfly-theta.conf

+LPGROUPS
+{
+   MODELNET_GRP
+   {
+      repetitions="864";
+# name of this lp changes according to the model
+      nw-lp="4";
+# these lp names will be the same for dragonfly-custom model
+      modelnet_dragonfly_custom="4";
+      modelnet_dragonfly_custom_router="1";
+   }
+}
+PARAMS
+{
+# packet size in the network 
+   packet_size="1024";
+   modelnet_order=( "dragonfly_custom","dragonfly_custom_router" );
+   # scheduler options
+   modelnet_scheduler="fcfs";
+# chunk size in the network (when chunk size = packet size, packets will not be
+# divided into chunks)
+   chunk_size="1024";
+   # modelnet_scheduler="round-robin";
+   # number of routers within each group
+   # this is dictated by the dragonfly configuration files
+   num_routers="96";
+   # number of groups in the network
+   num_groups="9";
+# buffer size in bytes for local virtual channels 
+   local_vc_size="8192";
+#buffer size in bytes for global virtual channels 
+   global_vc_size="16384";
+#buffer size in bytes for compute node virtual channels 
+   cn_vc_size="8192";
+#bandwidth in GiB/s for local channels 
+   local_bandwidth="5.25";
+# bandwidth in GiB/s for global channels 
+   global_bandwidth="4.7";
+# bandwidth in GiB/s for compute node-router channels 
+   cn_bandwidth="16.0";
+# ROSS message size 
+   message_size="584";
+# number of compute nodes connected to router, dictated by dragonfly config
+# file
+   num_cns_per_router="4";
+# number of global channels per router 
+   num_global_channels="12";
+# network config file for intra-group connections 
+   intra-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/intra-theta";
+# network config file for inter-group connections
+   inter-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/inter-theta";
+# routing protocol to be used 
+   routing="prog-adaptive";
+}

src/network-workloads/model-net-synthetic.c

@@ -26,7 +26,6 @@ static double arrival_time = 1000.0;
 static int num_servers_per_rep = 0;
 static int num_routers_per_grp = 0;
 static int num_nodes_per_grp = 0;
-static int num_nodes_per_cn = 0;
 static int num_groups = 0;
 static int num_nodes = 0;
 
@@ -202,7 +201,7 @@ static void handle_kickoff_event(
     memcpy(m_remote, m_local, sizeof(svr_msg));
     m_remote->svr_event_type = REMOTE;
 
-    assert(net_id == DRAGONFLY || net_id == DRAGONFLY_CUSTOM); /* only supported for dragonfly model right now. */
+    assert(net_id == DRAGONFLY); /* only supported for dragonfly model right now. */
     ns->start_ts = tw_now(lp);
     codes_mapping_get_lp_info(lp->gid, group_name, &group_index, lp_type_name, &lp_type_index, anno, &rep_id, &offset);
     int local_id = codes_mapping_get_lp_relative_id(lp->gid, 0, 0);
@@ -376,20 +375,19 @@ int main(
     g_tw_ts_end = s_to_ns(5 * 24 * 60 * 60);
     model_net_enable_sampling(sampling_interval, sampling_end_time);
 
-    if(net_id != DRAGONFLY && net_id != DRAGONFLY_CUSTOM)
+    if(net_id != DRAGONFLY)
     {
-	printf("\n The test works with dragonfly model configuration only! %d %d ", DRAGONFLY_CUSTOM, net_id);
+	printf("\n The test works with dragonfly model configuration only! ");
         MPI_Finalize();
         return 0;
     }
     num_servers_per_rep = codes_mapping_get_lp_count("MODELNET_GRP", 1, "server",
             NULL, 1);
     configuration_get_value_int(&config, "PARAMS", "num_routers", NULL, &num_routers_per_grp);
-    configuration_get_value_int(&config, "PARAMS", "num_groups", NULL, &num_groups);
-    configuration_get_value_int(&config, "PARAMS", "num_cns_per_router", NULL, &num_nodes_per_cn);
 
-    num_nodes = num_groups * num_routers_per_grp * num_nodes_per_cn;
-    num_nodes_per_grp = num_routers_per_grp * num_nodes_per_cn;
+    num_groups = (num_routers_per_grp * (num_routers_per_grp/2) + 1);
+    num_nodes = num_groups * num_routers_per_grp * (num_routers_per_grp / 2);
+    num_nodes_per_grp = num_routers_per_grp * (num_routers_per_grp / 2);
 
     assert(num_nodes);
 

src/networks/model-net/dragonfly.c

@@ -31,7 +31,6 @@
 #define TRACK_MSG -1
 #define PRINT_ROUTER_TABLE 1
 #define DEBUG 0
-#define USE_DIRECT_SCHEME 1
 #define MAX_STATS 65536
 
 #define LP_CONFIG_NM_TERM (model_net_lp_config_names[DRAGONFLY])
@@ -775,7 +774,6 @@ void router_setup(router_state * r, tw_lp * lp)
 //   printf("\n LP ID %d VC occupancy radix %d Router %d is connected to ", lp->gid, p->radix, r->router_id);
 #endif 
    //round the number of global channels to the nearest even number
-#if USE_DIRECT_SCHEME
        int first = r->router_id % p->num_routers;
        for(int i=0; i < p->num_global_channels; i++)
         {
@@ -790,29 +788,6 @@ void router_setup(router_state * r, tw_lp * lp)
             r->global_channel[i] = target_grp * p->num_routers + my_pos;
             first += p->num_routers;
         }
-#else
-   int router_offset = (r->router_id % p->num_routers) * 
-    (p->num_global_channels / 2) + 1;
-   for(int i=0; i < p->num_global_channels; i++)
-    {
-      if(i % 2 != 0)
-          {
-             r->global_channel[i]=(r->router_id + (router_offset * p->num_routers))%p->total_routers;
-             router_offset++;
-          }
-          else
-           {
-             r->global_channel[i]=r->router_id - ((router_offset) * p->num_routers);
-           }
-        if(r->global_channel[i]<0)
-         {
-           r->global_channel[i]=p->total_routers+r->global_channel[i]; 
-	 }
-#if DEBUG == 1
-    printf("\n channel %d ", r->global_channel[i]);
-#endif 
-    }
-#endif
 
 #if DEBUG == 1
    printf("\n");
@@ -895,35 +870,11 @@ tw_lpid getRouterFromGroupID(int dest_gid,
 		    int num_routers,
             int total_groups)
 {
-#if USE_DIRECT_SCHEME
   int dest = dest_gid;
   if(dest == total_groups - 1) {
       dest = src_gid;
   }
   return src_gid * num_routers + (dest % num_routers);
-#else
-  int group_begin = src_gid * num_routers;
-  int group_end = (src_gid * num_routers) + num_routers-1;
-  int offset = (dest_gid * num_routers - group_begin) / num_routers;
-  
-  if((dest_gid * num_routers) < group_begin)
-    offset = (group_begin - dest_gid * num_routers) / num_routers; // take absolute value
-  
-  int half_channel = num_routers / 4;
-  int index = (offset - 1)/(half_channel * num_routers);
-  
-  offset=(offset - 1) % (half_channel * num_routers);
-
-  // If the destination router is in the same group
-  tw_lpid router_id;
-
-  if(index % 2 != 0)
-    router_id = group_end - (offset / half_channel); // start from the end
-  else
-    router_id = group_begin + (offset / half_channel);
-
-  return router_id;
-#endif
 }	
 
 /*When a packet is sent from the current router and a buffer slot becomes available, a credit is sent back to schedule another packet event*/
@@ -2120,19 +2071,11 @@ get_next_stop(router_state * s,
 
       if(dest_lp == local_router_id)
       {
-#if USE_DIRECT_SCHEME
           int my_pos = s->group_id % s->params->num_routers;
           if(s->group_id == s->params->num_groups - 1) {
               my_pos = dest_group_id % s->params->num_routers;
           }
           dest_lp = dest_group_id * s->params->num_routers + my_pos;
-#else
-        for(int i=0; i < s->params->num_global_channels; i++)
-           {
-            if(s->global_channel[i] / s->params->num_routers == dest_group_id)
-                dest_lp=s->global_channel[i];
-          }
-#endif
       }
   codes_mapping_get_lp_id(lp_group_name, LP_CONFIG_NM_ROUT, s->anno, 0, dest_lp,
           0, &router_dest_id);
@@ -2166,20 +2109,12 @@ get_output_port( router_state * s,
 
      if(intm_grp_id != s->group_id)
       {
-#if USE_DIRECT_SCHEME
           int target_grp = intm_grp_id;
           if(target_grp == s->params->num_groups - 1) {
               target_grp = s->group_id;
           }
           output_port = s->params->num_routers + (target_grp) / 
                 s->params->num_routers;
-#else
-        for(int i=0; i < s->params->num_global_channels; i++)
-         {
-           if(s->global_channel[i] == local_router_id)
-             output_port = s->params->num_routers + i;
-          }
-#endif
       }
       else
        {