GitLab

Non-idle nodes are now fully respected.  Consistiently get string nid
lists out of this.  ValueError doesn't get raised if the attrs location
exists stradling a reservation (still in the queue, but not available
due to the reservation).

Fixing a situation where locations when set in a resrvation job causes
issues.

There was one further step needed for running jobs. Also fixing a
potential statefile issue with prior versions.

Nodes brought up in a disabled state at machine boot will not be
reported in ALPS.  This allows us to detect and recover when nodes are
warm-swapped back into service.

Update node state was resetting an admin down.  Added an additional flag
so we can differentiate between admin down and hardware down.

If a node is marked down with an admin command, then no matter what, it
will remain marked down.

Support for apkill added to kill user alps instnace in interactive jobs.
Kachina testing pending.

Resources for interactive jobs are now appropriately released.  There is
still a known issue with currently running aprun instances.  That will
be addressed in a further patch.

This would prevent re-reservation if not fixed.

Rereservations were broken for long (>5 min) startups.  This should
allow the CAPMC scripts to do their thing.

maxtotaljobs limit added.

This adds the limiter for maximum jobs overall running in queue.  Useful
for profiling machines with noisy network environments.  This also adds
output to cqadm for this information, and an entry in the cqadm manpage.

See merge request !10

This adds the limiter for maximum jobs overall running in queue.  Useful
for profiling machines with noisy network environments.  This also adds
output to cqadm for this information, and an entry in the cqadm manpage.

Multiple forkers

Support for multiple Forkers (i.e. multiple script hosts) for Cray systems.

See merge request !9

Well, we're at least back at original functionality.  Forkers are
automatically acquired and dispatch to at least the single forker case
appears to work.

The alps forker can now rename itself at runtime.  Will be needed to
identify multiple components.

The output redirect for consoles needs some work in the init script
still.

Fix 17 compact format pbs

This fixes the PBS file format so the nodes for cray systems are in a compact format.  Other systems should not have their records affected.

Some of this logic should probably be extended into the cluster systems themselves.

See merge request !8

This was a less trivial change than I thought.  Had to do this all in
the system component to avoid Cray handling logic leakage into other
components like cqm.

On restart, if cobalt was shutdown abruptly (like with a power failure
or a kill -9), there was a way to lose the forker child process of a
process group.  The process group would never finish cleaning up, and
the associated resources would keep being put into cleanup-pending by
the reserve_resources_until code.

Now the orphaned process group(s) are cleaned up automatically.  CQM
jobs that reference these should get back an error stating that the
underlying task no longer exists/cannot be found.

This circumstance should be rare in production (I hope), but  I could
see this scenario being triggered during abnormal operations (like a
facility power/cooling failure).

info.

Enh 13 cobalt reservations

This brings reservation support to Cray systems.

See merge request !7

The overlap check was failing.  Has been modified for cray systems such
that the check is entirely local with no call the the system component.
There is no reason for the remote information at this point due to no
possibility of node overlap.

There was bug that was counting active reservation nodes as 2 nodes for
the purposes of determining how many nodes were left in the
non-reservation queue.