GitLab

This reverts merge request !54

This reverts merge request !49

This reverts merge request !50

Slight bit of reorganization.  Also added some things to make the
presentation elsewhere in Cobalt kind of consistent with the rest of
Cray's stack.

In light of this bug, adding checks to make sure that we don't end up
accidentally adding in bad values to reservations again.

This was traced to a call that could cause a non-string key to be added
to the alps_reservation dictionary, resulting in a version of the
reservation with an integer jobid key and a second with a string jobid
key.  These should be keyed with strings.

Added as further mitigation a check to see if there is an integer
version of a key to clean.  If there is, then notify that it happened
and clean that one, too.

Triggering condition is an interactive job where the initial ALPS
reservation times out.

Making form of ALPS reserve/confirm messages consistient.
Adding a jobid to cleanup complete (we can do that on this system!)

This reverts merge request !43

Instead of enumerating all groups on the system and comparing, check members of specific queue groups. This change makes CQM compatible with sssd.conf: enumerate = False, and overall is doing less work.

This is a pretty common check so this should be a cheap optimization for
us.

HBMCACHEPCT explicitly named and no longer operating on a node list,
goiong straight to nid list.  Should save memory/time.

Adding in a better validtor to prevent issues with users typing bad
NUMA/MCDRAM modes.  Also, adding a default setting if none provided.

We need to log and end the task timers on both __rc_unknown and
__rc_success on __task_finalize.  The only outcome that doesn't
officially end a task is __rc_retry, as it will go into communication
retries at that point.

Only set the timers/emit messages once, despite retry attempts.

start.

Uncovered another bug while working on this, old behavior masked it, but
there was a way to get a block to ignore the scheduled flag as well.

A well (or poorly depending on how you look at it) qdel could cause
Cobalt to put a node into cleanup but never complete the cleanup due to
there being no ALPS backend reservation to clean up.  This would clear
if there were no jobs currently running, however, it would hang nodes
otherwise.

If the child fetch succeeds but cleanup fails, make sure we use the
intially fetched data, rahter than replacing it with the now potentially
lost child data.

After discussion the current algorithm for determining backfill time
needs to be replaced and needs to depend on which blocks are selected
for draining.  This is a commit for the current algorithm's optimistic
and pessimistic backfill modes.

New tests pending, but the optimistic mode backfiller does appear to be
working properly.  Old behavior is preserved and may be enabled by
setting the mode to pessimistic.

Old versions of the forker do not have use_stdout_string that can casue
_wait() to fail.  Getting out of this would require deleting the
statefile and restarting clean.

To prevent that, the startup is being modified to add those key
variables and initalizing them to being "unused".

There was the possiblity of losign a PID if child cleanup was
interrupted.  This ensures retries until the child process is actually
dead.