Commit 60ee5f73 authored by Philip Carns's avatar Philip Carns
Browse files

various misc. edits

parent b9fbd06f
......@@ -48,3 +48,7 @@ Once modified, the job script may be submitted as follows.
```
$ sbatch ./job.sbatch
```
Note that the job script explicitly multiple PSM2 environment variables that
are necessary for Mochi compatibility. Please copy these environment
variables to any other Mochi job scripts that you create for the Bebop platform.
......@@ -16,8 +16,9 @@ $ module swap PrgEnv-intel PrgEnv-gnu
$ module load gcc/9.3.0
```
You may want to `export CRAYPE_LINK_TYPE=dynamic` (e.g. in your `.bashrc` file)
to avoid problems with dynamic libraries when building programs.
You may also want to `export CRAYPE_LINK_TYPE=dynamic` (e.g. in your
`.bashrc` file) to prevent the compiler from attempting to use static
linking in conjunction with the dynamic libraries produced by Spack.
Networking
......@@ -42,18 +43,18 @@ Once modified, the job script may be submitted as follows.
$ qsub ./job.qsub
```
Note the use of a protection domain (pdomain): these are necessary if
you have multiple applications, started using distinct `aprun` calls,
and you want them to communicate via Mercury. Protection domains are
global, but user-specific (i.e. you won't be able to use a pdomain
setup by someone else, and you won't be able to use the same name
as someone else's pdomain). There is a limited number of pdomains
that can exist simultaneously, and they are not cleaned up when your
job terminates, so please make sure to delete your pdomain at the
end of your job.
Note the use of a protection domain (pdomain). Shared protection domains
enable processes to communicate with each other even when launched with
separate `aprun` commands. Protection domains are global, but user-specific
(i.e. you won't be able to use a pdomain setup by someone else, and you
won't be able to use the same name as someone else's pdomain). There is
a limited number of pdomains that can exist simultaneously, and they are
not cleaned up when your job terminates, so please make sure to delete
your pdomain at the end of your job.
Note that other Cray systems may use `DRC (Dynamic RDMA Credentials)` rather
than static protection domains for this purpose. The protection domain
configuration in this example is specific to the Theta system.
The `MPICH_GNI_NDREG_ENTRIES` environment variable should be set
to avoid conflicts between MPI and libfabric.
`HDF5_USE_FILE_LOCKING=FALSE` has been shown to be useful for some
workloads using HDF5 on Lustre.
......@@ -7,8 +7,9 @@
set -eu
# prevents interference between MPI and Mochi (libfabric) libraries
# when sharing GNI resources
export MPICH_GNI_NDREG_ENTRIES=1024
export HDF5_USE_FILE_LOCKING=FALSE
PDOMAIN=mypdomain
......
......@@ -3,17 +3,17 @@ Platform configurations for Mochi
This repository provides Spack configuration files, example job scripts, and
notes about building and running Mochi-based codes on various platforms.
Please refer to your platform of interest for more information.
Please refer to the subdirectory for your platform of interest for more
information.
Using spack.yaml files
----------------------
Each platform subdirectory in this repository provides a `spack.yaml` file.
Such a file is meant to fully describe a Spack environment, including some system-provided
A `spack.yaml` file fully describes a Spack environment, including system-provided
packages and compilers. It does so independently of any `compilers.yaml` or `packages.yaml`
files installed in `~/.spack`, hence preventing as much as possible the possible interference
with user-specific spack configurations.
files installed in `~/.spack`, thereby preventing interference with user-specific spack configurations as much as possible.
You may use `spack.yaml` files to create a
[Spack environment](https://spack.readthedocs.io/en/latest/environments.html) in
......@@ -27,7 +27,8 @@ $ . spack/share/spack/setup-env.sh
```
Remember that the second line needs to be executed every time you open a new
terminal.
terminal; it may be helpful to create an alias in your bashrc file as a
shortcut.
You will then need to clone `sds-repo`, which contains the Mochi packages.
......@@ -54,7 +55,7 @@ Then, execute the following command
$ spack env create myenv spack.yaml
```
Change to a directory outside the `platform-configurations` folders
Change to a directory outside of the `platform-configurations` folders
and activate the environment as follows.
```
......@@ -102,10 +103,13 @@ Should you want to contribute a `spack.yaml` for a particular machine,
please submit a merge request with it, and ensure the following.
* The `spack.yaml` file should contain the compiler(s) that have been tested
to be working with Mochi packages.
and confirmed to work with Mochi packages.
* The `spack.yaml` file should try to list system-provided packages,
in particular packages used for building (`cmake`, `autoconf`, etc.),
and relevant system-provided MPI implementations.
- Note that this must be done manually. Spack provides a `spack external
find` command that can be used to locate a subset of system packages,
but it does not populate the `spack.yaml` file.
* The `spack.yaml` file should contain the relevant variants for packages,
in particular the transport methods to use with `libfabric`.
* The path to the `spack.yaml` file should be of the form
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment