GitLab

This reverts merge request !54

This reverts merge request !49

This reverts merge request !50

Slight bit of reorganization.  Also added some things to make the
presentation elsewhere in Cobalt kind of consistent with the rest of
Cray's stack.

In light of this bug, adding checks to make sure that we don't end up
accidentally adding in bad values to reservations again.

This was traced to a call that could cause a non-string key to be added
to the alps_reservation dictionary, resulting in a version of the
reservation with an integer jobid key and a second with a string jobid
key.  These should be keyed with strings.

Added as further mitigation a check to see if there is an integer
version of a key to clean.  If there is, then notify that it happened
and clean that one, too.

Triggering condition is an interactive job where the initial ALPS
reservation times out.

Making form of ALPS reserve/confirm messages consistient.
Adding a jobid to cleanup complete (we can do that on this system!)

This reverts merge request !43

This is a pretty common check so this should be a cheap optimization for
us.

HBMCACHEPCT explicitly named and no longer operating on a node list,
goiong straight to nid list.  Should save memory/time.

Adding in a better validtor to prevent issues with users typing bad
NUMA/MCDRAM modes.  Also, adding a default setting if none provided.

A well (or poorly depending on how you look at it) qdel could cause
Cobalt to put a node into cleanup but never complete the cleanup due to
there being no ALPS backend reservation to clean up.  This would clear
if there were no jobs currently running, however, it would hang nodes
otherwise.

If the child fetch succeeds but cleanup fails, make sure we use the
intially fetched data, rahter than replacing it with the now potentially
lost child data.

There was the possiblity of losign a PID if child cleanup was
interrupted.  This ensures retries until the child process is actually
dead.

This could happen when the node goes down while a job is running,
causing the node to still show up in the job end_times.

In the end the buffer size had to be increased to avoid timing issues.
Added in further try-except safety checks to prevent system component
issues if this runs long agian.

Backfillng has an epsilon of 2 minutes by default.  This can be altered
in the cobalt config file.

There was a way to set up resrvations across disjoint queues that caused
one set of queues to ignore that a reservation was pending because the
reservation wasn't associated with that equivalence class.  This caused
forbidden locations to not be set.

Draining and backfilling are passing basic tests.  Need to add more test
cases to the automated suite and test corner cases around
queues/reservations/locations list.

Also need to add backfill time display to nodelist/nodeadm -l.

This should get rid of the bulk of the 1234567 exit statuses. Forces a
timeout.  The timeout goes away when the job is started.  This should
fix the process group initilization/start gap.