Commit 5fcd5f10 authored by Matthieu Dorier's avatar Matthieu Dorier

added instructions for running on Theta

parent 14f2549a
Deploying and running on Theta
==============================
This section specifically describes how to install
and run HEPnOS and HEPnOS programs on Theta.
Setting up spack
----------------
If you don't have spack already, you need to set it up.
.. code-block:: console
git clone https://github.com/spack/spack.git
. spack/share/spack/setup-env.sh
Note that the second line should be run everytime you log in
to bring spack into your PATH.
Now edit *~/.spack/cray/packages.yaml* with the following.
.. container:: toggle
.. container:: header
.. container:: btn btn-info
packages.yaml (show/hide)
.. literalinclude:: ../../examples/theta/packages.yaml
:language: yaml
Those are system packages that Spack won't need to install.
Setting up Mochi
----------------
Once spack is installed, you need to add the mochi repository.
.. code-block:: console
git clone https://xgitlab.cels.anl.gov/sds/sds-repo
Setting up a Spack environment
------------------------------
We will then create an environment called *my-hepnos-env* and
add HEPnOS to it.
.. code-block:: console
spack env create hepnos-env
spack env activate hepnos-env
Edit the *spack/var/spack/environments/hepnos-env/spack.yaml* file
so it looks like the following.
.. code-block:: yaml
spack:
specs:
- hepnos
view: true
repos:
- /gpfs/mira-home/<usernam>/sds-repo
packages:
libfabric:
variants: fabrics=gni
mercury:
variants: +udreg ~boostsys
Replace <username> by your username (or the full path to the cloned
sds-repo folder if you have cloned it somewhere else than in you home).
You may add more specs in this file if you need.
Now type :code:`spack install` to install HEPnOS.
Running HEPnOS
--------------
To run HEPnOS on Theta, the HEPnOS's YAML configuration file should
use *ofi+gni* in the address field.
Theta by default restricts communication between processes of distinct
MPI applications. To enable such connections, you need to create a
protection domain. This can be done as follows from the MOM nodes
(i.e. either inside the script you give to qsub, or when you are in
an interactive session).
.. code-block:: console
apmgr pdomain -c -u mydomain
Replace *mydomain* with a suitable name. This name should not conflict
with a name setup by someone else. Also the protection domain is persistent
across job submissions, so keep it available only if you need it later,
and if not, use the following command to destroy it once your job has completed.
.. code-block:: console
apmgr pdomain -r -u mydomain
HEPnOS can be started on a set of nodes (either from a qsub script or
in an interactive sessions) using the following command.
.. code-block:: console
. spack/share/spack/setup-env.sh
spack env activate hepnos-env
aprun -n X -N 1 -p mydomain hepnos-daemon <config.yaml> <client.yaml> &
Replace <config.yaml> by the HEPnOS configuration file and <client.yaml>
by the name of the file you want to create for clients to connect.
packages:
all:
providers:
mpi: [mpich, openmpi]
compiler: [gcc@8.3.0, intel@18.0.2.199, cce@8.7.3]
autoconf:
paths:
autoconf@2.69: /usr
buildable: False
automake:
paths:
automake@1.13.4: /usr
buildable: False
binutils:
paths:
binutils@2.31.1: /usr
buildable: False
bison:
paths:
bison@2.7: /usr
buildable: False
boost:
modules:
boost@1.64.0 arch=cray-cnl6-mic_knl: boost/gnu/1.64.0
buildable: False
bzip2:
paths:
bzip2@1.0.6: /usr
buildable: False
cmake:
modules:
cmake@3.14.5: cmake/3.14.5
buildable: False
coreutils:
paths:
coreutils@8.25: /usr
buildable: False
cppunit:
variants: cxxstd=14
findutils:
paths:
findutils@4.5.12: /usr
buildable: False
flex:
paths:
flex@2.5.37: /usr
buildable: False
gettext:
paths:
gettext@0.19.2: /usr
buildable: False
git:
paths:
git@2.12.3: /usr
buildable: False
hdf5:
modules:
hdf5@1.10.5.1: cray-hdf5/1.10.5.1
buildable: False
libtool:
paths:
libtool@2.4.2: /usr
buildable: False
m4:
paths:
m4@1.4.16: /usr
buildable: False
mpich:
modules:
mpich@7.7.3 arch=cray-cnl6-mic_knl: cray-mpich/7.7.3
buildable: False
perl:
paths:
perl@5.18.2: /usr
buildable: False
pkg-config:
paths:
pkg-config@0.29: /usr
buildable: False
tar:
paths:
tar@1.27.1: /usr
buildable: False
texinfo:
paths:
texinfo@4.13: /usr
buildable: False
xz:
paths:
xz@5.0.5: /usr
buildable: False
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment