GitLab

Init Postgres backend works; added "balsam which"; Can run sqlite/Postgres backends from "balsam dbserver"; Upped timeouts/queue depths for ZMQ Sqlite server

workers(nodes) available to Balsam; useful for running benchmarks

* timing capabilities built in to MPIEnsemble runner and MPIRunner
Now successful job runs should have an elapsed runtime in seconds
added to the field runtime_seconds

* Refactored utilites for automated testing and benchmarking

Job PK and version are properly updated with sqlite_client save method. test runner works with both save methods

* User explicitly creates balsam DB directory with "balsam init"
* User launches DB server with "balsam dbserver"
* user_settings is vastly simplified...moved all configuration logic
into django_config module
* Database is auto-configured on django startup; will connect to DB
server automatically if client

* user kill works even when a kill save() takes a long time in flight
  and some other processing happens first
* updated functional tests
* all tests passing on Theta, Cooley

all back into development branch.

Merge remote-tracking branch 'origin/theta-changes' into develop

Added test cases

concurrently; they block by queueing up at the DB-writer, which is a ZMQ
proxy to the Sqlite database

* dag.spawn_child now clones jobs in a more sane way; fixed bug related
to job_id being sent over the wire as None

* improved logging

* Fixed deserialize bug when job_id is None (for new jobs)

* added test case for concurrent BalsamJob insertions to database from a
single mpi4py applicaiton, where all ranks are concurrently calling
save() with new jobs

cli_commands: fixed small user error message bug
model saving: more robust retry: wait and check if save succeeded in
spite of the exception

An mpi4py test program with the Python sqlite3 wrapper has no trouble
with concurrent DB insertions from several tens of processes.  The issue
appears to be in the Django-sqlite3 wrapper.

This patch simply wraps save() in a loop that intercepts
OperationalError until the save is successful.

This reverts commit 74ce64c5