GitLab

This could happen when the node goes down while a job is running,
causing the node to still show up in the job end_times.

Backfillng has an epsilon of 2 minutes by default.  This can be altered
in the cobalt config file.

There was a way to set up resrvations across disjoint queues that caused
one set of queues to ignore that a reservation was pending because the
reservation wasn't associated with that equivalence class.  This caused
forbidden locations to not be set.

Draining and backfilling are passing basic tests.  Need to add more test
cases to the automated suite and test corner cases around
queues/reservations/locations list.

Also need to add backfill time display to nodelist/nodeadm -l.

This should get rid of the bulk of the 1234567 exit statuses. Forces a
timeout.  The timeout goes away when the job is started.  This should
fix the process group initilization/start gap.

Checking in fixes for find queue equivalence classes that impact
draining.  Drain-status-clear now working.  Stub for drain selection.

Mocked up by replicating node data in testing.

Make sure that an update cannot change this list midflight for a job.

Caller also holds this lock at this point in time.  The node_lock must
be reentrant!

Thanks Eric!  Duplicated nids are now avoided.

Fixes attrs location evading cobalt admin down on nodes.

Non-idle nodes are now fully respected.  Consistiently get string nid
lists out of this.  ValueError doesn't get raised if the attrs location
exists stradling a reservation (still in the queue, but not available
due to the reservation).

The apid fetch wasn't restricting itself to the actual ALPS reservation.
This was causing everything to get killed.

System component restart on the fly should be safe again.  We recover
the process groups properly now.  Found this while testing other changes
in the fix for aggressive cleanup.

Fixing a situation where locations when set in a resrvation job causes
issues.

There was one further step needed for running jobs. Also fixing a
potential statefile issue with prior versions.

Nodes brought up in a disabled state at machine boot will not be
reported in ALPS.  This allows us to detect and recover when nodes are
warm-swapped back into service.

Update node state was resetting an admin down.  Added an additional flag
so we can differentiate between admin down and hardware down.

If a node is marked down with an admin command, then no matter what, it
will remain marked down.

Support for apkill added to kill user alps instnace in interactive jobs.
Kachina testing pending.

Resources for interactive jobs are now appropriately released.  There is
still a known issue with currently running aprun instances.  That will
be addressed in a further patch.

Rereservations were broken for long (>5 min) startups.  This should
allow the CAPMC scripts to do their thing.

Well, we're at least back at original functionality.  Forkers are
automatically acquired and dispatch to at least the single forker case
appears to work.