Commit 8f92a75b authored by Misbah Mubarak's avatar Misbah Mubarak

Preparing beta version for new dragonfly model, modifications to routings,...

Preparing beta version for new dragonfly model, modifications to routings, adding documentation for generating config files, adding libtools to support C++ code compilation
parent 62564150
Nikhil Jain (UIUC)
Nikhil Jain, Abhinav Bhatele (LLNL)
- Improvements in credit-based flow control of CODES dragonfly and torus network models.
- Addition of direct scheme for setting up dragonfly network topology.
- Network configuration setup for custom dragonfly model.
- Topology generations scripts for custom dragonfly model.
- Bug reporter for CODES network models.
- Fat tree network setup and adaptive routing.
Jens Domke (U. of Dresden)
- Static routing in fat tree network model.
Xu Yang (IIT)
- Added support for running multiple application workloads with CODES MPI
......
......@@ -319,7 +319,7 @@ transparent message passing support between LPs, allowing the underlying
network LPs to do the work of routing while user LPs model their applications.
It consists of a number of both simple and complex network models as well as
configuration utilities and communication APIs. A somewhat stale overview is
also given at src/models/networks/model-net/doc/README.
also given at src/networks/model-net/doc/README.
= Components of model-net
......@@ -407,8 +407,8 @@ for the latency and bandwidth costs: "net_latency_ns_file" and
"net_bw_mbps_file".
More details about the models can be found at
src/models/networks/model-net/doc/README.simplenet.txt and
src/models/networks/model-net/doc/README.simplep2p.txt, respectively.
src/networks/model-net/doc/README.simplenet.txt and
src/networks/model-net/doc/README.simplep2p.txt, respectively.
== LogGP
......@@ -422,7 +422,7 @@ The only configuration entry the LogGP model requires is
configuration file.
For more details on gathering parameters for the LogGP model, as well as it's
usage and caveats, see the document src/models/model-net/doc/README.loggp.txt.
usage and caveats, see the document src/model-net/doc/README.loggp.txt.
== Torus
......@@ -435,7 +435,7 @@ performed in:
(PADS), 2014.
The configuration and model setup can be found at:
src/models/model-net/doc/README.torus.txt
src/model-net/doc/README.torus.txt
== Dragonfly
......@@ -443,7 +443,7 @@ The dragonfly model (model-net LP name: "dragonfly") is a network
topology that utilizes the concept of virtual routers to produce systems with
very high virtual radix out of network components with a lower radix. The
topology itself and the simulation model are both described in
src/models/networks/model-net/doc/README.dragonfly.txt.
src/networks/model-net/doc/README.dragonfly.txt.
cite).
The configuration parameters are a little trickier here, as additional LPs
......@@ -453,7 +453,7 @@ represents a physical router. At least one "dragonfly_router" LP must be
present in every LP group with a "modelnet_dragonfly" LP.
Further configuration and model setup can be found at
src/models/model-net/doc/README.dragonfly.txt.
src/model-net/doc/README.dragonfly-custom.txt.
= CODES example model
......
** Generating inter and intra group files for Edison and Theta Interconnects **:
- Edison network config files:
python gen-network-config.py link-edison.txt intra-edison inter-edison
- Theta network config files:
python gen-network-config.py theta intra-theta inter-theta
** Generating customizable dragonfly interconnects **:
mpicc connections_general.c -o connections_general
./connections_general g r c intra-file inter-file
--> g: number of groups in the network
--> r: number of router rows within a group
--> c: number of router columns within a group
--> intra-file: output files for intra-group connections
--> inter-file: output file for inter-group connections
- The scripts and code for translating existing topologies and generating
cray-style dragonfly topologies have been contributed by Nikhil Jain, Abhinav
Bhatele and Peer-Timo Breemer from LLNL.
- For details on cray XC dragonfly network topology, see the following paper:
@inproceedings{faanes2012cray,
title={Cray cascade: a scalable HPC system based on a Dragonfly network},
author={Faanes, Greg and Bataineh, Abdulla and Roweth, Duncan and Froese,
Edwin and Alverson, Bob and Johnson, Tim and Kopnick, Joe and Higgins, Mike
and Reinhard, James and others},
booktitle={Proceedings of the International Conference on High
Performance Computing, Networking, Storage and Analysis},
pages={103},
year={2012},
organization={IEEE Computer Society Press}
}
......@@ -21,16 +21,16 @@
int main(int argc, char **argv) {
if(argc < 3) {
printf("Corret usage: %s <num_g> <intra_file> <inter_file>", argv[0]);
printf("Correct usage: %s <num_g> <num_rows> <num_cols> <intra_file> <inter_file>", argv[0]);
exit(0);
}
int g = atoi(argv[1]);
int r = 6;
int c = 16;
int r = atoi(argv[2]);
int c = atoi(argv[3]);
FILE *intra = fopen(argv[2], "wb");
FILE *inter = fopen(argv[3], "wb");
FILE *intra = fopen(argv[4], "wb");
FILE *inter = fopen(argv[5], "wb");
int router = 0;
int green = 0, black = 1;
......
......@@ -175,6 +175,7 @@ bin_PROGRAMS += src/workload/codes-workload-dump
bin_PROGRAMS += src/network-workloads/model-net-mpi-replay
bin_PROGRAMS += src/network-workloads/model-net-dumpi-traces-dump
bin_PROGRAMS += src/network-workloads/model-net-synthetic
bin_PROGRAMS += src/network-workloads/model-net-synthetic-custom-dfly
bin_PROGRAMS += src/network-workloads/model-net-synthetic-slimfly
src_workload_codes_workload_dump_SOURCES = \
......@@ -182,6 +183,7 @@ src_workload_codes_workload_dump_SOURCES = \
src_network_workloads_model_net_mpi_replay_SOURCES = src/network-workloads/model-net-mpi-replay.c
src_network_workloads_model_net_synthetic_SOURCES = src/network-workloads/model-net-synthetic.c
src_network_workloads_model_net_synthetic_custom_dfly_SOURCES = src/network-workloads/model-net-synthetic-custom-dfly.c
src_network_workloads_model_net_synthetic_slimfly_SOURCES = src/network-workloads/model-net-synthetic-slimfly.c src/network-workloads/dummy.C
src_network_workloads_model_net_dumpi_traces_dump_SOURCES = src/network-workloads/model-net-dumpi-traces-dump.c src/network-workloads/dummy.C
......
LPGROUPS
{
MODELNET_GRP
{
repetitions="1440";
# name of this lp changes according to the model
server="4";
# these lp names will be the same for dragonfly-custom model
modelnet_dragonfly_custom="4";
modelnet_dragonfly_custom_router="1";
}
}
PARAMS
{
# packet size in the network
packet_size="1024";
modelnet_order=( "dragonfly_custom","dragonfly_custom_router" );
# scheduler options
modelnet_scheduler="fcfs";
# chunk size in the network (when chunk size = packet size, packets will not be
# divided into chunks)
chunk_size="1024";
# modelnet_scheduler="round-robin";
# number of routers within each group
# this is dictated by the dragonfly configuration files
num_routers="96";
# number of groups in the network
num_groups="15";
# buffer size in bytes for local virtual channels
local_vc_size="8192";
#buffer size in bytes for global virtual channels
global_vc_size="16384";
#buffer size in bytes for compute node virtual channels
cn_vc_size="8192";
#bandwidth in GiB/s for local channels
local_bandwidth="5.25";
# bandwidth in GiB/s for global channels
global_bandwidth="4.7";
# bandwidth in GiB/s for compute node-router channels
cn_bandwidth="8.0";
# ROSS message size
message_size="584";
# number of compute nodes connected to router, dictated by dragonfly config
# file
num_cns_per_router="4";
# number of global channels per router
num_global_channels="8";
# network config file for intra-group connections
intra-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/intra-edison";
# network config file for inter-group connections
inter-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/inter-edison";
# routing protocol to be used
routing="adaptive";
}
......@@ -2,32 +2,53 @@ LPGROUPS
{
MODELNET_GRP
{
repetitions="1440";
nw-lp="1";
modelnet_dragonfly_custom="1";
repetitions="3200";
# name of this lp changes according to the model
nw-lp="4";
# these lp names will be the same for dragonfly-custom model
modelnet_dragonfly_custom="4";
modelnet_dragonfly_custom_router="1";
}
}
PARAMS
{
packet_size="512";
# packet size in the network
packet_size="1024";
modelnet_order=( "dragonfly_custom","dragonfly_custom_router" );
# scheduler options
modelnet_scheduler="fcfs";
chunk_size="256";
# chunk size in the network (when chunk size = packet size, packets will not be
# divided into chunks)
chunk_size="1024";
# modelnet_scheduler="round-robin";
num_routers="96";
num_groups="15";
local_vc_size="2048";
global_vc_size="8192";
cn_vc_size="1024";
# number of routers within each group
# this is dictated by the dragonfly configuration files
num_routers="128";
# number of groups in the network
num_groups="25";
# buffer size in bytes for local virtual channels
local_vc_size="8192";
#buffer size in bytes for global virtual channels
global_vc_size="16384";
#buffer size in bytes for compute node virtual channels
cn_vc_size="8192";
#bandwidth in GiB/s for local channels
local_bandwidth="5.25";
# bandwidth in GiB/s for global channels
global_bandwidth="4.7";
cn_bandwidth="5.25";
# bandwidth in GiB/s for compute node-router channels
cn_bandwidth="8.0";
# ROSS message size
message_size="584";
num_cns_per_router="1";
num_global_channels="4";
intra-group-connections="intracray2";
inter-group-connections="intercray2";
# number of compute nodes connected to router, dictated by dragonfly config
# file
num_cns_per_router="4";
# number of global channels per router
num_global_channels="8";
# network config file for intra-group connections
intra-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/intra-custom";
# network config file for inter-group connections
inter-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/inter-custom";
# routing protocol to be used
routing="prog-adaptive";
}
LPGROUPS
{
MODELNET_GRP
{
repetitions="1440";
# name of this lp changes according to the model
nw-lp="4";
# these lp names will be the same for dragonfly-custom model
modelnet_dragonfly_custom="4";
modelnet_dragonfly_custom_router="1";
}
}
PARAMS
{
# packet size in the network
packet_size="1024";
modelnet_order=( "dragonfly_custom","dragonfly_custom_router" );
# scheduler options
modelnet_scheduler="fcfs";
# chunk size in the network (when chunk size = packet size, packets will not be
# divided into chunks)
chunk_size="1024";
# modelnet_scheduler="round-robin";
# number of routers within each group
# this is dictated by the dragonfly configuration files
num_routers="96";
# number of groups in the network
num_groups="15";
# buffer size in bytes for local virtual channels
local_vc_size="8192";
#buffer size in bytes for global virtual channels
global_vc_size="16384";
#buffer size in bytes for compute node virtual channels
cn_vc_size="8192";
#bandwidth in GiB/s for local channels
local_bandwidth="5.25";
# bandwidth in GiB/s for global channels
global_bandwidth="4.7";
# bandwidth in GiB/s for compute node-router channels
cn_bandwidth="8.0";
# ROSS message size
message_size="584";
# number of compute nodes connected to router, dictated by dragonfly config
# file
num_cns_per_router="4";
# number of global channels per router
num_global_channels="8";
# network config file for intra-group connections
intra-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/intra-edison";
# network config file for inter-group connections
inter-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/inter-edison";
# routing protocol to be used
routing="adaptive";
}
LPGROUPS
{
MODELNET_GRP
{
repetitions="864";
# name of this lp changes according to the model
nw-lp="4";
# these lp names will be the same for dragonfly-custom model
modelnet_dragonfly_custom="4";
modelnet_dragonfly_custom_router="1";
}
}
PARAMS
{
# packet size in the network
packet_size="1024";
modelnet_order=( "dragonfly_custom","dragonfly_custom_router" );
# scheduler options
modelnet_scheduler="fcfs";
# chunk size in the network (when chunk size = packet size, packets will not be
# divided into chunks)
chunk_size="1024";
# modelnet_scheduler="round-robin";
# number of routers within each group
# this is dictated by the dragonfly configuration files
num_routers="96";
# number of groups in the network
num_groups="9";
# buffer size in bytes for local virtual channels
local_vc_size="8192";
#buffer size in bytes for global virtual channels
global_vc_size="16384";
#buffer size in bytes for compute node virtual channels
cn_vc_size="8192";
#bandwidth in GiB/s for local channels
local_bandwidth="5.25";
# bandwidth in GiB/s for global channels
global_bandwidth="4.7";
# bandwidth in GiB/s for compute node-router channels
cn_bandwidth="16.0";
# ROSS message size
message_size="584";
# number of compute nodes connected to router, dictated by dragonfly config
# file
num_cns_per_router="4";
# number of global channels per router
num_global_channels="12";
# network config file for intra-group connections
intra-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/intra-theta";
# network config file for inter-group connections
inter-group-connections="/Users/mmubarak/Documents/software_development/dragonfly-cray/codes/scripts/gen-cray-topo/inter-theta";
# routing protocol to be used
routing="prog-adaptive";
}
This diff is collapsed.
......@@ -26,7 +26,6 @@ static double arrival_time = 1000.0;
static int num_servers_per_rep = 0;
static int num_routers_per_grp = 0;
static int num_nodes_per_grp = 0;
static int num_nodes_per_cn = 0;
static int num_groups = 0;
static int num_nodes = 0;
......@@ -202,7 +201,7 @@ static void handle_kickoff_event(
memcpy(m_remote, m_local, sizeof(svr_msg));
m_remote->svr_event_type = REMOTE;
assert(net_id == DRAGONFLY || net_id == DRAGONFLY_CUSTOM); /* only supported for dragonfly model right now. */
assert(net_id == DRAGONFLY); /* only supported for dragonfly model right now. */
ns->start_ts = tw_now(lp);
codes_mapping_get_lp_info(lp->gid, group_name, &group_index, lp_type_name, &lp_type_index, anno, &rep_id, &offset);
int local_id = codes_mapping_get_lp_relative_id(lp->gid, 0, 0);
......@@ -376,20 +375,19 @@ int main(
g_tw_ts_end = s_to_ns(5 * 24 * 60 * 60);
model_net_enable_sampling(sampling_interval, sampling_end_time);
if(net_id != DRAGONFLY && net_id != DRAGONFLY_CUSTOM)
if(net_id != DRAGONFLY)
{
printf("\n The test works with dragonfly model configuration only! %d %d ", DRAGONFLY_CUSTOM, net_id);
printf("\n The test works with dragonfly model configuration only! ");
MPI_Finalize();
return 0;
}
num_servers_per_rep = codes_mapping_get_lp_count("MODELNET_GRP", 1, "server",
NULL, 1);
configuration_get_value_int(&config, "PARAMS", "num_routers", NULL, &num_routers_per_grp);
configuration_get_value_int(&config, "PARAMS", "num_groups", NULL, &num_groups);
configuration_get_value_int(&config, "PARAMS", "num_cns_per_router", NULL, &num_nodes_per_cn);
num_nodes = num_groups * num_routers_per_grp * num_nodes_per_cn;
num_nodes_per_grp = num_routers_per_grp * num_nodes_per_cn;
num_groups = (num_routers_per_grp * (num_routers_per_grp/2) + 1);
num_nodes = num_groups * num_routers_per_grp * (num_routers_per_grp / 2);
num_nodes_per_grp = num_routers_per_grp * (num_routers_per_grp / 2);
assert(num_nodes);
......
This diff is collapsed.
......@@ -31,7 +31,6 @@
#define TRACK_MSG -1
#define PRINT_ROUTER_TABLE 1
#define DEBUG 0
#define USE_DIRECT_SCHEME 1
#define MAX_STATS 65536
#define LP_CONFIG_NM_TERM (model_net_lp_config_names[DRAGONFLY])
......@@ -775,7 +774,6 @@ void router_setup(router_state * r, tw_lp * lp)
// printf("\n LP ID %d VC occupancy radix %d Router %d is connected to ", lp->gid, p->radix, r->router_id);
#endif
//round the number of global channels to the nearest even number
#if USE_DIRECT_SCHEME
int first = r->router_id % p->num_routers;
for(int i=0; i < p->num_global_channels; i++)
{
......@@ -790,29 +788,6 @@ void router_setup(router_state * r, tw_lp * lp)
r->global_channel[i] = target_grp * p->num_routers + my_pos;
first += p->num_routers;
}
#else
int router_offset = (r->router_id % p->num_routers) *
(p->num_global_channels / 2) + 1;
for(int i=0; i < p->num_global_channels; i++)
{
if(i % 2 != 0)
{
r->global_channel[i]=(r->router_id + (router_offset * p->num_routers))%p->total_routers;
router_offset++;
}
else
{
r->global_channel[i]=r->router_id - ((router_offset) * p->num_routers);
}
if(r->global_channel[i]<0)
{
r->global_channel[i]=p->total_routers+r->global_channel[i];
}
#if DEBUG == 1
printf("\n channel %d ", r->global_channel[i]);
#endif
}
#endif
#if DEBUG == 1
printf("\n");
......@@ -895,35 +870,11 @@ tw_lpid getRouterFromGroupID(int dest_gid,
int num_routers,
int total_groups)
{
#if USE_DIRECT_SCHEME
int dest = dest_gid;
if(dest == total_groups - 1) {
dest = src_gid;
}
return src_gid * num_routers + (dest % num_routers);
#else
int group_begin = src_gid * num_routers;
int group_end = (src_gid * num_routers) + num_routers-1;
int offset = (dest_gid * num_routers - group_begin) / num_routers;
if((dest_gid * num_routers) < group_begin)
offset = (group_begin - dest_gid * num_routers) / num_routers; // take absolute value
int half_channel = num_routers / 4;
int index = (offset - 1)/(half_channel * num_routers);
offset=(offset - 1) % (half_channel * num_routers);
// If the destination router is in the same group
tw_lpid router_id;
if(index % 2 != 0)
router_id = group_end - (offset / half_channel); // start from the end
else
router_id = group_begin + (offset / half_channel);
return router_id;
#endif
}
/*When a packet is sent from the current router and a buffer slot becomes available, a credit is sent back to schedule another packet event*/
......@@ -2120,19 +2071,11 @@ get_next_stop(router_state * s,
if(dest_lp == local_router_id)
{
#if USE_DIRECT_SCHEME
int my_pos = s->group_id % s->params->num_routers;
if(s->group_id == s->params->num_groups - 1) {
my_pos = dest_group_id % s->params->num_routers;
}
dest_lp = dest_group_id * s->params->num_routers + my_pos;
#else
for(int i=0; i < s->params->num_global_channels; i++)
{
if(s->global_channel[i] / s->params->num_routers == dest_group_id)
dest_lp=s->global_channel[i];
}
#endif
}
codes_mapping_get_lp_id(lp_group_name, LP_CONFIG_NM_ROUT, s->anno, 0, dest_lp,
0, &router_dest_id);
......@@ -2166,20 +2109,12 @@ get_output_port( router_state * s,
if(intm_grp_id != s->group_id)
{
#if USE_DIRECT_SCHEME
int target_grp = intm_grp_id;
if(target_grp == s->params->num_groups - 1) {
target_grp = s->group_id;
}
output_port = s->params->num_routers + (target_grp) /
s->params->num_routers;
#else
for(int i=0; i < s->params->num_global_channels; i++)
{
if(s->global_channel[i] == local_router_id)
output_port = s->params->num_routers + i;
}
#endif
}
else
{
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment