[MARGO_ENABLE_PROFILING] Issue with profile file generation when Thallium is used
@robl Discovered an issue where the
margo-gen-profile script fails when trying to generate a profile summary for a simple Thallium example: https://mochi.readthedocs.io/en/latest/thallium/15_finalize.html
Upon further inspection, it seems that the problem is the following: Under some conditions, Thallium appears to "short" certain RPC calls by bypassing the regular Margo call chain. As a result, the profile files tell us that for certain RPCs, a client made an RPC call but there is no trace of the same call on the server side from the generated .csv profile files. For the above example, this happens for the "print" and the "hello" calls. @mdorier confirmed that this is because Thallium replaces a few of the Margo functionalities.
From an implementation perspective, Margo has to see the invocation of the "__margo_internal_pre_wrapper_hooks" and the "margo_respond" calls in order for the server to register and profile the RPC. Please find attached sample profile files generated by @robl. Simply run $MARGO_INSTALL/bin/margo-gen-profile in the directory containing these two files to see the issue. If the issue is fixed, this script runs to completion and generates a profile.pdf file.
In order to profile a Thallium/Margo application, please set MARGO_ENABLE_PROFILING=1. profile-hodor-25611.csv