We might want to add functions to start and stop the progress loop.

When calling margo_init on a server and specifying that the progress loop runs on an additional execution stream, we actually make the progress loop start before RPCs have been registered. Consequently, a client may try to contact the server and get as a response that the RPC id doesn't map to any RPC handler. I have had that happening in my kNN code and I suspect this was the reason.

The second problem with the fact that margo_init starts the progress loop is that we don't protect the margo parameters: if you call margo_set_param and set a parameter right at the moment this parameter is being used by the progress loop, you can end up with strange behaviors.

I would advocate for a model where we do margo_init followed by an explicit margo_start (maybe with some margo_set_param in between), and also provide a margo_stop to allow another thread to explicitly pause the progress loop.

What do you think?