Multiple clients with Margo 0.2
I am using Margo as part of a preloading library, where I require two separate Margo clients that run with different NA plugins. One is running with the NA SM (shared memory) plugin, and the other with the NA BMI plugin with TCP (for now). Both are started to run within the caller's thread context. This would work with Margo 0.1 because users had to deal with initializing Argobots themselves.
Margo 0.2 starts Argobots transparantly to the user, resulting in multiple Argobots initializations and sets the caller ES to idle with ABT_snoozer each time (with
ABT_snoozer_xstream_self_set()). Specifically, the latter action appears to be the culprit as the second call of
ABT_snoozer_xstream_self_set() (due to the second Margo client initialization) returns the errorcode 29 (
[../../src/stream.c:811] ABT_xstream_set_main_sched: 29). This aborts the whole client initialization process and
I've attached a small patch (margo_0.2_multiple_clients.patch) that highlights the issue and solves it by simply checking if Argobots is already initialized and, if so, skips both mentioned actions. However, I am questioning if this is the correct way of using multiple clients in one application.