update Summit information to specify mlx5_0 domain

ORNL/Summit/README.md

@@ -18,9 +18,13 @@ The `spack.yaml` file in this directory uses gcc 9.1.0.
 Networking
 ----------
 
-Mercury should use the `ofi+verbs;ofi_rxm` transport method in Mercury.
-Libfabric should be compiled with the `+verbs` and `+rxm` variants.
+Mercury should use the `verbs` transport method in Mercury, which
+translates to `ofi+verbs;ofi_rxm` internally.  You must also specify the
+libfabric domain when initializing Mercury by using `verbs://mlx5_0`
+as the local address. This will insure that libfabric
+uses the correct default port on the network card.
 
+Libfabric should be compiled with the `+verbs` and `+rxm` variants.
 
 Job management
 --------------

ORNL/Summit/job.lfs

@@ -10,5 +10,8 @@ export FI_MR_CACHE_MAX_COUNT=0
 # use shared recv context in RXM; should improve scalability
 export FI_OFI_RXM_USE_SRX=1
 
-jsrun -a 1 -n 2 -r 1 -c ALL_CPUS ./margo-p2p-latency -i 100000 -n "ofi+verbs;ofi_rxm://"
-jsrun -a 1 -n 2 -r 1 -c ALL_CPUS ./margo-p2p-bw -x 1048576 -n "ofi+verbs;ofi_rxm://" -c 1 -D 20
+# Note that we explicitly specify the libfabric domain of "mlx5_0" on
+# Summit. Otherwise libfabric and/or libibverbs may select a default port
+# that does not work out of the box.
+jsrun -a 1 -n 2 -r 1 -c ALL_CPUS ./margo-p2p-latency -i 100000 -n "verbs://mlx5_0"
+jsrun -a 1 -n 2 -r 1 -c ALL_CPUS ./margo-p2p-bw -x 1048576 -n "verbs://mlx5_0" -c 1 -D 20