README.md 5.6 KB
Newer Older
1
# Margo
2

3
Margo is a utility library built atop Mercury that simplifies RPC service
Philip Carns's avatar
tweak  
Philip Carns committed
4
development by providing bindings that can service concurrent operations while
5 6 7 8 9 10 11 12 13
hiding the complexity of callback functions and progress loops.

Margo does this by leveraging the Argobots user-level threading system to
transparently and efficiently context switch when functions are waiting
on the completion of Margo operations.  Other user-level threads can
therefore continue to make progress while one or more user-level threads
are blocked on network resources.  This approach combines the performance
advantages of Mercury's native event-driven execution model with the
progamming simplicity of a multi-threaded execution model.
14 15 16

See the following for more details about Mercury and Argobots: 

17
* https://mercury-hpc.github.io/
18
* https://collab.mcs.anl.gov/display/ARGOBOTS/Argobots+Home
19

20
Note that Margo should be compatible with any Mercury transport (NA plugin).  The documentation assumes the use of the NA SM (shared memory) plugin that is built into Mercury for simplicity.  This plugin is only valid for communication between processes on a single node.  See [Using Margo with other Mercury NA plugins](##using-margo-with-other-mercury-na-plugins) for information on other configuration options.
21

22 23 24
##  Dependencies

* mercury  (git clone --recurse-submodules https://github.com/mercury-hpc/mercury.git)
25 26
* argobots (git clone https://github.com/pmodels/argobots.git)
* abt-snoozer (git clone https://xgitlab.cels.anl.gov/sds/abt-snoozer)
27 28
* libev (e.g libev-dev package on Ubuntu or Debian)

29 30
### Recommended Mercury build options

31 32
* Mercury must be compiled with -DMERCURY_USE_BOOST_PP:BOOL=ON to enable the
  Boost preprocessor macros for encoding.
33 34 35
* -DMERCURY_USE_CHECKSUMS:BOOL=OFF disables automatic checksumming of all
  Mercury RPC messages.  This reduces latency by removing a layer of
  integrity checking on communication.
36 37 38
* Mercury should be compiled with -DMERCURY_USE_SELF_FORWARD:BOOL=ON in order to enable
  fast execution path for cases in which a Mercury service is linked into the same
  executable as the client
39

40 41 42 43 44
Example Mercury compilation:

```
mkdir build
cd build
45
cmake -DMERCURY_USE_SELF_FORWARD:BOOL=ON -DMERCURY_USE_CHECKSUMS:BOOL=OFF \
46 47 48 49 50
 -DBUILD_TESTING:BOOL=ON -DMERCURY_USE_BOOST_PP:BOOL=ON \
 -DCMAKE_INSTALL_PREFIX=/home/pcarns/working/install \
 -DBUILD_SHARED_LIBS:BOOL=ON -DCMAKE_BUILD_TYPE:STRING=Debug ../
```

51 52 53 54 55 56 57
## Building

Example configuration:

    ../configure --prefix=/home/pcarns/working/install \
        PKG_CONFIG_PATH=/home/pcarns/working/install/lib/pkgconfig \
        CFLAGS="-g -Wall"
Philip Carns's avatar
Philip Carns committed
58

59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105
## Running examples

The examples subdirectory contains:

* margo-example-client.c: an example client
* margo-example-server.c: an example server
* my-rpc.[ch]: an example RPC definition

The following example shows how to execute them.  Note that when the server starts it will display the address that the client can use to connect to it.


```
$ examples/margo-example-server na+sm://
# accepting RPCs on address "na+sm://13367/0"
Got RPC request with input_val: 0
Got RPC request with input_val: 1
Got RPC request with input_val: 2
Got RPC request with input_val: 3
Got RPC request to shutdown

$ examples/margo-example-client na+sm://13367/0
ULT [0] running.
ULT [1] running.
ULT [2] running.
ULT [3] running.
Got response ret: 0
ULT [0] done.
Got response ret: 0
ULT [1] done.
Got response ret: 0
ULT [2] done.
Got response ret: 0
ULT [3] done.
```

The client will issue 4 concurrent RPCs to the server and wait for them to
complete.

## Running tests

`make check`

## Using Margo with the other NA plugins

You can use either the CCI NA plugin or BMI NA plugin to use either the CCI or BMI library for remote communication.  See the [Mercury documentation](http://mercury-hpc.github.io/documentation/) for details and status.

### CCI
106

107 108
Add the -DNA_USE_CCI:BOOL=ON option to the Mercury configuration.

109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131
You must set the CCI_CONFIG environment variable to point to a valid CCI
configuration file.  You can use the following example and un-comment the
appropriate section for the transport that you wish to use.  Note that there
is no need to specify a port; Mercury will dictate a port for CCI to use if
needed.

```
[mercury]
# use this example for TCP
transport = tcp
interface = lo  # switch this to eth0 or an external hostname for non-localhost use

## use this example instead for shared memory
# transport = sm

## use this example instead for InfiniBand
# transport = verbs
# interface = ib0
```

You must then use addresses appropriate for your transport at run time when
executing Margo examples.  Examples for server "listening" addresses:

132 133 134
* cci+tcp://3344 # for TCP/IP, listening on port 3344
* cci+verbs://3344 # for InfiniBand, listening on port 3344
* cci+sm://1/1 # for shared memory, listening on CCI SM address 1/1
135 136 137

Examples for clients to specify to attach to the above:

138 139
* cci+tcp://localhost:3344 # for TCP/IP, assuming localhost use
* cci+verbs://192.168.1.78:3344 # for InfiniBand, note that you *must* use IP
140
  address rather than hostname
141
* cci+sm:///tmp/cci/sm/`hostname`/1/1 # note that this is a full path to local
142 143 144
  connection information.  The last portion of the path should match the
  address specified above

145
### BMI
146

147 148 149 150
Add the -DNA_USE_BMI:BOOL=ON option to the Mercury configuration.  You may
also need to specify
-DBMI_INCLUDE_DIR:PATH=/home/pcarns/working/install/include and -DBMI_LIBRARY:FILEPATH=/home/pcarns/working/install/lib/libbmi.a (adjusting the paths as appropriate for your system).

151
We do not recommend using any BMI methods besides TCP.  It's usage is very similar to the CCI/TCP examples above, except that "bmi+" should be substituted for "cci+".