Update node state was resetting an admin down.  Added an additional flag
so we can differentiate between admin down and hardware down.

If a node is marked down with an admin command, then no matter what, it
will remain marked down.

This would prevent re-reservation if not fixed.

Rereservations were broken for long (>5 min) startups.  This should
allow the CAPMC scripts to do their thing.

maxtotaljobs limit added.

This adds the limiter for maximum jobs overall running in queue.  Useful
for profiling machines with noisy network environments.  This also adds
output to cqadm for this information, and an entry in the cqadm manpage.

See merge request !10

This adds the limiter for maximum jobs overall running in queue.  Useful
for profiling machines with noisy network environments.  This also adds
output to cqadm for this information, and an entry in the cqadm manpage.

Multiple forkers

Support for multiple Forkers (i.e. multiple script hosts) for Cray systems.

See merge request !9

Well, we're at least back at original functionality.  Forkers are
automatically acquired and dispatch to at least the single forker case
appears to work.

The alps forker can now rename itself at runtime.  Will be needed to
identify multiple components.

The output redirect for consoles needs some work in the init script
still.

Fix 17 compact format pbs

This fixes the PBS file format so the nodes for cray systems are in a compact format.  Other systems should not have their records affected.

Some of this logic should probably be extended into the cluster systems themselves.

See merge request !8

This was a less trivial change than I thought.  Had to do this all in
the system component to avoid Cray handling logic leakage into other
components like cqm.

On restart, if cobalt was shutdown abruptly (like with a power failure
or a kill -9), there was a way to lose the forker child process of a
process group.  The process group would never finish cleaning up, and
the associated resources would keep being put into cleanup-pending by
the reserve_resources_until code.

Now the orphaned process group(s) are cleaned up automatically.  CQM
jobs that reference these should get back an error stating that the
underlying task no longer exists/cannot be found.

This circumstance should be rare in production (I hope), but  I could
see this scenario being triggered during abnormal operations (like a
facility power/cooling failure).

info.

Enh 13 cobalt reservations

This brings reservation support to Cray systems.

See merge request !7

The overlap check was failing.  Has been modified for cray systems such
that the check is entirely local with no call the the system component.
There is no reason for the remote information at this point due to no
possibility of node overlap.

There was bug that was counting active reservation nodes as 2 nodes for
the purposes of determining how many nodes were left in the
non-reservation queue.

Need to add display on nodelist for active reservation queues, and
appear to have some circumstances where the forbidden list isn't being
populated.

This is an interrim checkin of reservation handling.  This code is not
yet functional.  Saving prior to branch switch.

Fix 16 node cleanup

Merging in a change to fix a bug where nodes were not cleaning up as long as any reservations were on the system.  This also fixes Cobalt ignoring node roles.  It will now only try to schedule on batch nodes.

See merge request !6

New statuses in 1.7 are supported.  Now correctly marking node non-idle
for alps-interactive (effectively a 'down' state).  Removed redundant
release check.

Resources weren't actually exititng the cleanup state when there were
other resrervations on the system.  The check to mark nodes idle was not
actually ocurring when a reservation existed to mark nodes as idle..

Enh 14 use system reservednodes

Smaller system query added.  RESERVENODES support will be added in a later ticket.  Also has fixes from first encounters with Kachina.

See merge request !5

This should significantly reduce the overhead of the system inventory
for updating state.  Also gets memory statuses.

Need to add dynamic attribute update for running systems.

We can now get and properly display node attributes via the system type
query.

Cray's documentation on what depth and nppn do isn't all that clear.
Apparenlty this arrangement will actually reserve proper numbers of
nodes.

Full allocation now works reliably.

The alps_script_forker has to call BASIL, too.  Make sure they all point
to the same one.

Fix 12 scheduler loop performance

Fixes for improving the performance of scheduling loop and improving performance of clients during state update and job scheduling.

See merge request !4

This is an issue I saw with higher job counts.  This works on a
per-queue basis and will prevent an attempt at placement if Cobalt
already knows there aren't enough available nodes for a job.