Commit 8bf1829a authored by Misbah Mubarak's avatar Misbah Mubarak

Dragonfly-custom model beta version: adding documentation, fixing model-net test program

parent 8f92a75b
** Generating inter and intra group files for Edison and Theta Interconnects **:
- Edison network config files:
python gen-network-config.py link-edison.txt intra-edison inter-edison
- Theta network config files:
python gen-network-config.py theta intra-theta inter-theta
python edison.py link-edison.txt intra-edison inter-edison
[intra-edison and inter-edison are the intra-group and inter-group network
configuration files required by the simulation. The python script
translates Edison's network configuration into a file format that can be fed
into the simulation.]
** Generating customizable dragonfly interconnects **:
......
------- Network Layout ---------
- The network layout for the custom dragonfly model is similar to Cray XC
systems. Each group has X number of routers arranged in rows and columns.
There are all-to-all connections among routers within the same row and same
column. If a packet is sent to a router that is within the same group but
lies on a different row and column, it will be first sent to an intermediate
router that has a direct connection to the destination router.
- There can be multiple global channels between two groups as specified by the
network configuration files. Currently, Edison has 12 global channels
connecting any two groups in the network.
- For more details on the Cray dragonfly topology, see "Cray cascade: a
scalable HPC system based on a Dragonfly network" by Greg Faanes et al.
in Supercomputing 2012.
------- Network Setup ---------
- Edison Configuration: The network configuration file from Cray Edison system
at NERSC can be used to setup the dragonfly network topology. The
configuration has 5,760 nodes, 1440 routers and 15 groups. The instructions
on how to geneate the network configuration files for Edison can be found at:
codes/scripts/gen-cray-topo/README.txt
- Custom Configuration: The model can be configured with an arbitrary number of
groups and number of routers within a group. For instructions on how to
generate a custom network configuration, see
codes/scripts/gen-cray-topo/README.txt
------- Routing Protocols ------
- Minimal: Within a group, a minimal route will traverse three intermediate
hops at the maximum (a source router, an intermediate router if source and
destination routers do not share the same row or column and the destination
router). Across groups, a minimal route will choose the shortest possible
connection to the destination group. A global minimal route will traverse a
maximum of 6 router hops (including source and destination).
- Non-Minimal: Within a group, a non-minimal route will direct the packets to a
randomly selected router first. A global non-minimal route will involve
routing to a randomly selected intermediate router from the network first. A
global non-minimal route can traverse up to 11 router hops (including source
and destination).
- Local adaptive: Local adaptive routing takes a non-minimal route within a
group if it detects congestion on the minimal route (queues on minimal port
are used to detect congestion).
- Global Adaptive: Global adaptive routing takes a global non-minimal route if
it detects congestion on the minimal route (queues on minimal port are used
to detect congestion).
- Progressive Adaptive: Progressive adaptive routing re-evaluates the decision
to take a minimal or a non-minimal route as long as the packet stays in the
source group. If a non-minimal route is decided at some point in the source
group, the decision is no more re-evaluated.
-------- Running simulations --------
- Synthetic Traffic Patterns:
[With custom dragonfly network having 6,400 network nodes, 1600 routers and
20 groups. Each group has 80 routers arranged in a 20x4 matrix]
./bin/model-net-synthetic-custom-dfly --sync=1 -- ../src/network-workloads/conf/modelnet-synthetic-custom.conf
[With edison dragonfly network have 5,702 network nodes, 1440 routers and 15
groups. Each group has 96 routers arranged in 6x16 matrix.]
mpirun -np 4 ./bin/model-net-synthetic-custom-dfly --sync=3 -- ../src/network-workloads/conf/modelnet-synthetic-edison.conf
- Design Forward Network traces:
[With Multigrid network trace having 125 ranks]
./bin/model-net-mpi-replay --sync=1 --disable_compute=1
--workload_type="dumpi"
--workload_file=../../../df_traces/Multigrid/MultiGrid_C_n125_dumpi/dumpi-2014.03.06.23.48.13-
--num_net_traces=125 --
../src/network-workloads/conf/modelnet-test-dragonfly-edison.conf
--------- Debugging Tips ------------
- Set DUMP_CONNECTIONS debugging option to see the detailed local and global
channel connectivity of routers in src/networks/model-net/dragonfly-custom.C
......@@ -20,7 +20,7 @@
#include <vector>
#include <map>
#define DUMP_CONNECTIONS 0
#define DUMP_CONNECTIONS 1
#define CREDIT_SIZE 8
#define DFLY_HASH_TABLE_SIZE 262144
......
......@@ -24,16 +24,18 @@
#include "codes/configuration.h"
#include "codes/lp-type-lookup.h"
#define SVR_LP_NM "server"
#define NUM_REQS 2 /* number of requests sent by each server */
#define PAYLOAD_SZ 4096 /* size of simulated data payload, bytes */
static int net_id = 0;
static int num_routers = 0;
static int num_servers = 0;
static int offset = 2;
/* whether to pull instead of push */
static int do_pull = 0;
static int num_routers_per_rep = 0;
static int num_servers_per_rep = 0;
static int lps_per_rep = 0;
......@@ -58,7 +60,6 @@ struct svr_state
int local_recvd_count; /* number of local messages received */
tw_stime start_ts; /* time that we started sending requests */
tw_stime end_ts; /* time that we ended sending requests */
tw_lpid svr_rel_id; /* relative ID of the server */
};
struct svr_msg
......@@ -174,9 +175,26 @@ int main(
net_id = *net_ids;
free(net_ids);
num_servers = codes_mapping_get_lp_count("MODELNET_GRP", 0, SVR_LP_NM,
num_servers = codes_mapping_get_lp_count("MODELNET_GRP", 0, "server",
NULL, 1);
if(net_id == DRAGONFLY)
{
strcpy(router_name, "modelnet_dragonfly_router");
}
if(net_id == SLIMFLY)
{
strcpy(router_name, "slimfly_router");
}
if(net_id == SLIMFLY || net_id == DRAGONFLY)
{
num_routers = codes_mapping_get_lp_count("MODELNET_GRP", 0,
router_name, NULL, 1);
offset = 1;
}
if(lp_io_prepare("modelnet-test", LP_IO_UNIQ_SUFFIX, &handle, MPI_COMM_WORLD) < 0)
{
return(-1);
......@@ -214,11 +232,11 @@ static void svr_init(
memset(ns, 0, sizeof(*ns));
ns->svr_rel_id = codes_mapping_get_lp_relative_id(lp->gid, 0, 0);
/* each server sends a dummy event to itself that will kick off the real
* simulation
*/
//printf("\n Initializing servers %d ", (int)lp->gid);
/* skew each kickoff event slightly to help avoid event ties later on */
kickoff_time = g_tw_lookahead + tw_rand_unif(lp->rng);
......@@ -328,9 +346,24 @@ static void handle_kickoff_event(
/* record when transfers started on this server */
ns->start_ts = tw_now(lp);
int dest_id = (ns->svr_rel_id + 1) % num_servers;
num_servers_per_rep = codes_mapping_get_lp_count("MODELNET_GRP", 1,
"server", NULL, 1);
num_routers_per_rep = codes_mapping_get_lp_count("MODELNET_GRP", 1,
router_name, NULL, 1);
lps_per_rep = num_servers_per_rep * 2 + num_routers_per_rep;
int opt_offset = 0;
int total_lps = num_servers * 2 + num_routers;
if(net_id == DRAGONFLY && (lp->gid % lps_per_rep == num_servers_per_rep - 1))
opt_offset = num_servers_per_rep + num_routers_per_rep; /* optional offset due to dragonfly mapping */
if(net_id == SLIMFLY && (lp->gid % lps_per_rep == num_servers_per_rep -1))
opt_offset = num_servers_per_rep + num_routers_per_rep;
/* each server sends a request to the next highest server */
int dest_id = (lp->gid + offset + opt_offset)%total_lps;
if (do_pull){
m->ret = model_net_pull_event(net_id, "test", dest_id, PAYLOAD_SZ, 0.0,
sizeof(svr_msg), (const void*)m_remote, lp);
......@@ -416,7 +449,15 @@ static void handle_ack_event(
/* safety check that this request got to the right server */
// printf("\n m->src %d lp->gid %d ", m->src, lp->gid);
tw_lpid dest_id = codes_mapping_get_lpid_from_relative(m->src, NULL, SVR_LP_NM, NULL, 0);
int opt_offset = 0;
if(net_id == DRAGONFLY && (lp->gid % lps_per_rep == num_servers_per_rep - 1))
opt_offset = num_servers_per_rep + num_routers_per_rep; /* optional offset due to dragonfly mapping */
if(net_id == SLIMFLY && (lp->gid % lps_per_rep == num_servers_per_rep -1))
opt_offset = num_servers_per_rep + num_routers_per_rep;
tw_lpid dest_id = (lp->gid + offset + opt_offset)%(num_servers*2 + num_routers);
/* in the "pull" case, src should actually be self */
if (do_pull){
......@@ -469,6 +510,15 @@ static void handle_req_event(
/* safety check that this request got to the right server */
// printf("\n m->src %d lp->gid %d ", m->src, lp->gid);
int opt_offset = 0;
if(net_id == DRAGONFLY && (m->src % lps_per_rep == num_servers_per_rep - 1))
opt_offset = num_servers_per_rep + num_routers_per_rep; /* optional offset due to dragonfly mapping */
if(net_id == SLIMFLY && (m->src % lps_per_rep == num_servers_per_rep -1))
opt_offset = num_servers_per_rep + num_routers_per_rep;
assert(lp->gid == (m->src + offset + opt_offset)%(num_servers*2 + num_routers));
ns->msg_recvd_count++;
/* send ack back */
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment