GitLab

The initial setting of 1 second is too close to the update interval,
making this test fail when there is no actual problem due to jitter
between threads.  Extending this to 3 seconds should allow this to
reliably progress through the excessive reboot failure sequence.

The reservation would have gotten marked as dying even if the apids
hadn't been harvested.  Also, the data for the BASIL check was not
constructed correctly and while the right thing was happening, in that
an appication_id with a non-BASIL cmd was being resturned, the test
wasn't correct.  It's been fixed now.

Eric Pershey authored Mar 07, 2018 and Eric Pershey committed Mar 26, 2018

Wrapped Proxy errors with sanatize password.
Cleaned up logging spelling mistakes.
Added extract_traceback to format the exceptions
Added get_current_thread_identifier to help identify threads
Added get_caller to help identify the caller.
Added cray_messaging to aid in debugging and allow importing.
Fixed a bug that would allow retrying of a fork.
Added debugging tools of slp_hammer.py and slp_nail.py
Updated the simulator to simulate background tasks.

queue is no longer needed for cleanup, and the docstrings should
reflect the changed behavior.

These are the test cases for the bug that precipitated this hunt.  These
should largely be degenerate with the other single-equivalence cases
(because they are now single-equivalence cases), but do things like
make sure that drains are getting correctly cleared and times are
getting correctly reset.

After discussions, the current find_queue_equivalence_classes for this
system really only complicates the codebase for very little actual gain.
After this, the system will have only one equivalence class at all times
consisting of all active queues assigned to nodes and all active
reservations.

This simplification allows us to ensure that find_job_location only gets
called twice, once for reservations, which ignore drain times, and then
immediately after for the normal "production" queue jobs, which do set
drain times.  In both cases we can just clear drain times across the
machine.

In addition to testing (and more tests coming for the case that caused
this examination to begin with), we know that this works, as any system
with a queue or set of overlapping queues across all resources on the
machine forms a single equivalence class under the old code.

This can happen from some of the XMLRPC calls.

In light of this bug, adding checks to make sure that we don't end up
accidentally adding in bad values to reservations again.

Attrs is showing up more often due to the code to allow filters and the
validator to set attributes on a job.

start.

There was a change in the call and a behavior change to not try to
redrain blocks.  This was what the originally intended behavior was.

After discussion the current algorithm for determining backfill time
needs to be replaced and needs to depend on which blocks are selected
for draining.  This is a commit for the current algorithm's optimistic
and pessimistic backfill modes.

New tests pending, but the optimistic mode backfiller does appear to be
working properly.  Old behavior is preserved and may be enabled by
setting the mode to pessimistic.