Merge branch 'carns/editing' into 'master'

various edits

See merge request !2

ANL/Bebop/README.md

@@ -48,3 +48,7 @@ Once modified, the job script may be submitted as follows.
 ```
 $ sbatch ./job.sbatch
 ```
+
+Note that the job script explicitly multiple PSM2 environment variables that
+are necessary for Mochi compatibility.  Please copy these environment
+variables to any other Mochi job scripts that you create for the Bebop platform.

ANL/Theta/README.md

@@ -16,8 +16,9 @@ $ module swap PrgEnv-intel PrgEnv-gnu
 $ module load gcc/9.3.0
 ```
 
-You may want to `export CRAYPE_LINK_TYPE=dynamic` (e.g. in your `.bashrc` file)
-to avoid problems with dynamic libraries when building programs.
+You may also want to `export CRAYPE_LINK_TYPE=dynamic` (e.g. in your
+`.bashrc` file) to prevent the compiler from attempting to use static
+linking in conjunction with the dynamic libraries produced by Spack.
 
 
 Networking
@@ -42,18 +43,18 @@ Once modified, the job script may be submitted as follows.
 $ qsub ./job.qsub
 ```
 
-Note the use of a protection domain (pdomain): these are necessary if
-you have multiple applications, started using distinct `aprun` calls,
-and you want them to communicate via Mercury. Protection domains are
-global, but user-specific (i.e. you won't be able to use a pdomain
-setup by someone else, and you won't be able to use the same name
-as someone else's pdomain). There is a limited number of pdomains
-that can exist simultaneously, and they are not cleaned up when your
-job terminates, so please make sure to delete your pdomain at the
-end of your job.
+Note the use of a protection domain (pdomain).  Shared protection domains
+enable processes to communicate with each other even when launched with
+separate `aprun` commands.  Protection domains are global, but user-specific
+(i.e. you won't be able to use a pdomain setup by someone else, and you
+won't be able to use the same name as someone else's pdomain). There is
+a limited number of pdomains that can exist simultaneously, and they are
+not cleaned up when your job terminates, so please make sure to delete
+your pdomain at the end of your job.
+
+Note that other Cray systems may use `DRC (Dynamic RDMA Credentials)` rather
+than static protection domains for this purpose.  The protection domain
+configuration in this example is specific to the Theta system.
 
 The `MPICH_GNI_NDREG_ENTRIES` environment variable should be set
 to avoid conflicts between MPI and libfabric.
-	
-`HDF5_USE_FILE_LOCKING=FALSE` has been shown to be useful for some
-workloads using HDF5 on Lustre.

ANL/Theta/job.qsub

@@ -7,8 +7,9 @@
 
 set -eu
 
+# prevents interference between MPI and Mochi (libfabric) libraries
+# when sharing GNI resources
 export MPICH_GNI_NDREG_ENTRIES=1024
-export HDF5_USE_FILE_LOCKING=FALSE
 
 PDOMAIN=mypdomain
 

README.md

@@ -3,17 +3,17 @@ Platform configurations for Mochi
 
 This repository provides Spack configuration files, example job scripts, and
 notes about building and running Mochi-based codes on various platforms.
-Please refer to your platform of interest for more information.
+Please refer to the subdirectory for your platform of interest for more
+information.
 
 
 Using spack.yaml files
 ----------------------
 
 Each platform subdirectory in this repository provides a `spack.yaml` file.
-Such a file is meant to fully describe a Spack environment, including some system-provided
+A `spack.yaml` file fully describes a Spack environment, including system-provided
 packages and compilers. It does so independently of any `compilers.yaml` or `packages.yaml`
-files installed in `~/.spack`, hence preventing as much as possible the possible interference
-with user-specific spack configurations.
+files installed in `~/.spack`, thereby preventing interference with user-specific spack configurations as much as possible.
 
 You may use `spack.yaml` files to create a
 [Spack environment](https://spack.readthedocs.io/en/latest/environments.html) in
@@ -27,7 +27,8 @@ $ . spack/share/spack/setup-env.sh
 ```
 
 Remember that the second line needs to be executed every time you open a new
-terminal.
+terminal; it may be helpful to create an alias in your bashrc file as a
+shortcut.
 
 You will then need to clone `sds-repo`, which contains the Mochi packages.
 
@@ -54,7 +55,7 @@ Then, execute the following command
 $ spack env create myenv spack.yaml
 ```
 
-Change to a directory outside the `platform-configurations` folders
+Change to a directory outside of the `platform-configurations` folders
 and activate the environment as follows.
 
 ```
@@ -102,10 +103,13 @@ Should you want to contribute a `spack.yaml` for a particular machine,
 please submit a merge request with it, and ensure the following.
 
 *  The `spack.yaml` file should contain the compiler(s) that have been tested
-   to be working with Mochi packages.
+   and confirmed to work with Mochi packages.
 *  The `spack.yaml` file should try to list system-provided packages,
    in particular packages used for building (`cmake`, `autoconf`, etc.),
    and relevant system-provided MPI implementations.
+   - Note that this must be done manually.  Spack provides a `spack external
+     find` command that can be used to locate a subset of system packages,
+     but it does not populate the `spack.yaml` file.
 *  The `spack.yaml` file should contain the relevant variants for packages,
    in particular the transport methods to use with `libfabric`.
 *  The path to the `spack.yaml` file should be of the form