Commit 35a1372f authored by Rob Latham's avatar Rob Latham

first draft of summit onboarding

parent 39e64fcd
# Mochi Boot Camp: Hands-on
## Logging on to Summit
Summit requires two-factor authentication. Hopefully you have a token from Oak
Ridge. If you do not, and you are reading this on 6 February at the boot camp
for the first time, you will have to find another machine to work on.
You'll log in with your pin + one-time pass code.
Make a directory for this bootcamp:
mkdir ~/bootcamp
cd ~/bootcamp
git clone
## Installing spack
The easiest way to download and compile Mochi components is via the [Spack
package manager]( Begin by cloning the spack repository
from github, and then adding spack to your environment:
cd ~/bootcamp
git clone
cd spack
. ~/bootcamp/spack/share/spack/
At this point, the `spack` command line tool is available to you, but we
need just a few more steps to configure it ideally for this envionment.
Summit has a minimal base environment, and relies on `modules`
( to populate the environment.
We'll load a recent gcc compiler and teach spack about it:
[robl@login1]~% module load gcc/9.1.0
[robl@login1]~% spack compiler find
==> Added 1 new compiler to /home/robl/.spack/linux/compilers.yaml
==> Compilers are defined in the following files:
Other compilers, such as Intel, IBM's XL, or PGI compilers might work for some
or all of the Mochi components, but we know gcc-9 supports the language
features used by any of our components.
## Adding the Mochi software repository to Spack
Many of the Mochi software components have not yet been upstreamed to the
Spack package manager. They are available as a separate software repository
that can be added to spack with the `spack repo add` command:
[robl@login1]~% cd ~/bootcamp
[robl@login1]~/bootcamp% git clone
robl@login1]~/bootcamp% cd sds-repo
robl@login1]~/bootcamp/sds-repo% spack repo add sds-repo
[carns@jlselogin2 bootcamp]$
==> Added repo with namespace 'sds'.
## Customizing Spack for ORNL/Summit environment
At this point you can install any Mochi software component, but it will
likely download and compile more packages than are strictly necessary, which
is time consuming. At this point we will install a `packages.yaml` file
that customizes Spack by informing it of system packages that it should
reuse, and specifies a subset of network transports to use for Mochi.
We have provided a pre-configured `packages.yaml` file for this purpose in
the JLSE environment. It primarily does the following:
* informs Spack to use already available software for certain common system
packages (Spack by default will build these packages itself)
* Configure the Mercury RPC package to use the Infiniband 'verbs' interface for our
external network fabric package 'libfabric'
You can activate this configuration for your account by doing the following:
cp ~/bootcamp/mochi-boot-camp/ecp-am-2020/sessions/hands-on/packages.yaml ~/.spack/linux/
At this point you are ready to install and run Mochi software!
## General spack usage for package management
The following are the most important commands to know:
* `spack spec <package>` to see what will be installed if you were to
install it (including dependencies and version numbers)
* `spack install <package>` to install a package
* `spack load -r <package>` to load the package into your environment
* `module list` to observe what modules you have loaded
## Setting up your profile to retain Spack and compiler settings
There are two critical commands that you will want to either run every time
you log into a Summit node, or else add to your ~/.bashrc file so that they
are performed automatically. We recommend the latter to save time:
cat ~/bootcamp/mochi-boot-camp/ecp-am-2020/sessions/hands-on/bashrc.mochi.summit >> ~/.bashrc
Now when you log into Summit moving forward you will have the correct compiler
and Spack command line tools available in your environment.
## Installing your first Mochi components
Run the following to download, compile, and install Margo:
spack install margo
This will take a few minutes and will install all of the necessary
dependencies, including Mercury and Argobots. You can now load these
packages by running:
spack load -r margo
... and inspect to confirm that they are present in your environment with:
module list
## Compiling an example Mochi code
cd ~/bootcamp/mochi-boot-camp/ecp-am-2020/sessions/hands-on/sum
The above example is a very slightly modified (to use Infiniband instead
of TCP for communication) copy of the [Sending arguments, returning
values]( example
from the [Mochi Readthedocs
This will compile a simple client and server program, linked
against margo. If you inspect the Makfile you will see pkg-config commands
that are used to find the correct CFLAGS and LDFLAGS for the build.
## Running an interactive job on Summit compute nodes
Summit uses the LSF job scheduler plus some job managment utilites specific to
summit. One requests an allocation of nodes with `bsub` and runs a program in
that allocation with `jsrun`
To get an interactive allocation for 15 minutes:
bsub -Is -W 0:15 -nnodes 2 -P CSC332 $SHELL
You'll need to replace `-P CSC332` with the name of whichever project you're using.
The job scheduler will drop you into an interactive login on one of the
"monitor" nodes. Don't run jobs here directly. Instead, we'll use `jsrun`.
Note that we asked for two nodes from the scheduler. We'll run one service on
one node in the background, then run the client on the other.
First, start the server. The job will inherit your environment, so if you
loaded modules and built the client and server, things should work ok.
cd ~/bootcamp/ecp-am-2020/sessions/hands-on/sum
jsrun -n 1 -r 1 -g ALL_GPUS ./server &
Next, run the client, passing it the address string
jsrun -n 1 -r 1 -g ALL_GPUS ./client ofi+verbs://blah
## Additional resources
# Mochi bootcamp 2020 environment for
# no need to set compiler paths: spack extracted path information from the
# environment when user ran 'spack compiler find'
. ~/bootcamp/spack/share/spack/
compiler: [gcc@9.1.0, xl]
mpi: [spectrum-mpi, mpich]
pkgconfig: [pkg-config]
spectrum-mpi@ spectrum-mpi/
buildable: False
openssl@1.0.2k: /usr
buildable: False
# spack issue : system cmake
# (also built by spack) will confuse other cmake-using packages
# buildable: false
# modules:
# cmake@3.15.2: cmake/3.15.2
autoconf@2.69: /usr
buildable: False
automake@1.16.1: automake/1.16.1
buildable: False
ucx@1.5.1: /usr
buildable: False
libnl@3.3.0: libnl/3.3.0
buildable: False
rdma-core@20: /usr
buildable: False
findutils@4.5.11: /usr
buildable: False
libxml2@2.9.1: /usr
buildable: False
variants: +mpi
variants: ~boostsys
# the 'mlx' provider for libfabric is not maintained
variants: fabrics=verbs,rxm,mrail
variants: +cereal
variants: netmod=ucx device=ch4
CFLAGS += `pkg-config --cflags margo`
LDFLAGS += `pkg-config --libs margo`
all:: client server
client.o: client.c types.h
$(CC) $(CFLAGS) -c client.c
client: client.o
$(CC) client.o -o client $(LDFLAGS)
server.o: server.c types.h
$(CC) $(CFLAGS) -c server.c
server: server.o
$(CC) server.o -o server $(LDFLAGS)
rm -f client server client.o server.o
#include <assert.h>
#include <stdio.h>
#include <margo.h>
#include "types.h"
int main(int argc, char** argv)
if(argc != 2) {
fprintf(stderr,"Usage: %s <server address>\n", argv[0]);
margo_instance_id mid = margo_init("verbs:", MARGO_CLIENT_MODE, 0, 0);
hg_id_t sum_rpc_id = MARGO_REGISTER(mid, "sum", sum_in_t, sum_out_t, NULL);
hg_addr_t svr_addr;
margo_addr_lookup(mid, argv[1], &svr_addr);
int i;
sum_in_t args;
for(i=0; i<4; i++) {
args.x = 42+i*2;
args.y = 42+i*2+1;
hg_handle_t h;
margo_create(mid, svr_addr, sum_rpc_id, &h);
margo_forward(h, &args);
sum_out_t resp;
margo_get_output(h, &resp);
printf("Got response: %d+%d = %d\n", args.x, args.y, resp.ret);
margo_addr_free(mid, svr_addr);
return 0;
#include <assert.h>
#include <stdio.h>
#include <margo.h>
#include "types.h"
typedef struct {
int max_rpcs;
int num_rpcs;
} server_data;
static void sum(hg_handle_t h);
int main(int argc, char** argv)
margo_instance_id mid = margo_init("verbs:", MARGO_SERVER_MODE, 0, 0);
server_data svr_data = {
.max_rpcs = 4,
.num_rpcs = 0
hg_addr_t my_address;
margo_addr_self(mid, &my_address);
char addr_str[128];
size_t addr_str_size = 128;
margo_addr_to_string(mid, addr_str, &addr_str_size, my_address);
printf("Server running at address %s\n", addr_str);
hg_id_t rpc_id = MARGO_REGISTER(mid, "sum", sum_in_t, sum_out_t, sum);
margo_register_data(mid, rpc_id, &svr_data, NULL);
return 0;
static void sum(hg_handle_t h)
hg_return_t ret;
sum_in_t in;
sum_out_t out;
margo_instance_id mid = margo_hg_handle_get_instance(h);
const struct hg_info* info = margo_get_info(h);
server_data* svr_data = (server_data*)margo_registered_data(mid, info->id);
ret = margo_get_input(h, &in);
assert(ret == HG_SUCCESS);
out.ret = in.x + in.y;
printf("Computed %d + %d = %d\n",in.x,in.y,out.ret);
ret = margo_respond(h, &out);
assert(ret == HG_SUCCESS);
ret = margo_free_input(h, &in);
assert(ret == HG_SUCCESS);
ret = margo_destroy(h);
assert(ret == HG_SUCCESS);
svr_data->num_rpcs += 1;
if(svr_data->num_rpcs == svr_data->max_rpcs) {
#ifndef PARAM_H
#define PARAM_H
#include <mercury.h>
#include <mercury_macros.h>
/* We use the Mercury macros to define the input
* and output structures along with the serialization
* functions.
MERCURY_GEN_PROC(sum_out_t, ((int32_t)(ret)))
Markdown is supported
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment