Recently, the following pattern has caused me headaches:

• id = margo_init(...)
• initialize subsystems with id, including registering RPCs
• process RPCs from others

The problem that comes up from this approach is a race between other folks calling the subsystem RPCs and that subsystem's initialization. Once margo_init is called, the process is open for business.

A few ways to prevent this:

• call sleep(X) on the RPC callers :)
• restructure code such that initialization of subsystems doesn't require a margo id. Not exactly ideal.
• add the ability to defer kicking off the progress loop until we're ready for it to run (assuming that a dedicated progress thread is being used).

My preference is the third option, as it seems cleanest from a user perspective, though I'm open to suggestions.