Some of the CODES models specifically the MPI workload replay layer uses a stack to push data structures that need to be retrieved in the reverse handler. The models would push the objects on the stack expecting that the rollbacks would happen in the right order but ROSS may execute reverse events out of order. This may cause the model to exhibit non-deterministic behavior that would perturb the model statistics by a small degree. In the MPI replay layer, this problem can show up with a small difference in the original statistics. In some rare cases, it would lead to an unmatched send/receive error.
There has been preliminary discussion to allow ROSS to execute reverse events in the expected order, that should resolve this issue.