GitLab

After discussions, the current find_queue_equivalence_classes for this
system really only complicates the codebase for very little actual gain.
After this, the system will have only one equivalence class at all times
consisting of all active queues assigned to nodes and all active
reservations.

This simplification allows us to ensure that find_job_location only gets
called twice, once for reservations, which ignore drain times, and then
immediately after for the normal "production" queue jobs, which do set
drain times.  In both cases we can just clear drain times across the
machine.

In addition to testing (and more tests coming for the case that caused
this examination to begin with), we know that this works, as any system
with a queue or set of overlapping queues across all resources on the
machine forms a single equivalence class under the old code.

This can happen from some of the XMLRPC calls.

In light of this bug, adding checks to make sure that we don't end up
accidentally adding in bad values to reservations again.

This could happen when the node goes down while a job is running,
causing the node to still show up in the job end_times.

Backfillng has an epsilon of 2 minutes by default.  This can be altered
in the cobalt config file.

There was a way to set up resrvations across disjoint queues that caused
one set of queues to ignore that a reservation was pending because the
reservation wasn't associated with that equivalence class.  This caused
forbidden locations to not be set.

Draining and backfilling are passing basic tests.  Need to add more test
cases to the automated suite and test corner cases around
queues/reservations/locations list.

Also need to add backfill time display to nodelist/nodeadm -l.

Checking in fixes for find queue equivalence classes that impact
draining.  Drain-status-clear now working.  Stub for drain selection.

Thanks Eric!  Duplicated nids are now avoided.

Fixes attrs location evading cobalt admin down on nodes.

Non-idle nodes are now fully respected.  Consistiently get string nid
lists out of this.  ValueError doesn't get raised if the attrs location
exists stradling a reservation (still in the queue, but not available
due to the reservation).