GitLab

Refactored diff calculation code to work without needing changes in
coolr module (Patch for Commit 36401a84)

[fix] Fixes `cmd listen --filter` for power messages.

See merge request !46

[feature] Adds a --filter option to `cmd` for pretty-printing.

See merge request !44

The option accepts a message type and prints the values in the csv
format:
msgtype, time, payload if the --filter option is recognized
msgtype, time otherwise
The print is force-flushed on stdout.
This switches CI to the "refactored" CI identifier at argotests/tests

Integration power

See merge request !45

the gitlab-ci.yml file now points to argotest/gitlab/basic.yml on
master.

Fixes for cmd listen on power

Closes #26

See merge request !42

This commits simplifies the CI setup for this repository, offloading
all complexity to the "argotest" repository.

This includes renaming "progress" to performance in argo_perf_wrapper.
There are two distincts keywords in the messaging layer: "performance"
for all things related to hardware, and "progress", for all things
relating to the application.

The option is called uuid, not container

Downstream refactor

See merge request !41

These use libnrm/downstream-refactor and should be updated once that
branch is merged.

Doesn't work with the new downstream API.

Replace the downstream API handling by the new messaging layer. Not that
we don't have a clean way to deal with dynamic concurrency control using
this API, so we disable the handling of it for now.

Add downstream RPC client/server classes that are the same as the
upstream ones.

This is part of a series of changes to downstream to allow for more
reliable communications between the daemon and applications. At this
time, the daemon never replies, so the RPC_REQ is basically used as a
way to publish events to the daemon.

Add a config option to specify the location of the PMPI LD_PRELOAD
library available in libnrm. This should make it easier to use this
library.

Resolve "Integration test version pinning."

Closes #25

See merge request !40

This commit pins the integration tests. This is done in the .integration.nix
file, which in turns uses the derivations from the 'argotest'
repository. These pin argopkgs - which itself pins nixpkgs, so
everything should be completely reproducible.

The integration target provisions the dependencies using nix-shell and
runs the basic 'hello world' container test.

Resolve "Bad relative path handling between command line client and daemon"

Closes #8

See merge request !38

This is not as good as passing part of the manifest options forward,
but it still fixes some of the practical problems when using the
components together. The code makes sure that the manifests exists,
though.

[feature] commandline arguments, config file management.

See merge request !37

Added integration tests to the `.gitlab-ci.yaml` file. These tests
run any runner tagged "integration". They depend on the 'argotest'
repository master branch being able to run the tests, which
uses 'argopkgs' repository. The runner must be Nix
enabled. This therefore currently uses a rolling release of the
integration tests.

`cmd` now sends a container kill message to the upstream api and exits
whenever it receives SIGINT, via C-c for instance.

Fixing a bug introduced by the 'progress-report' branch in a recent
previous commit. The process object is the result of a tornado spawn, so
the call has to be slightly different than what was there.

This commit adds a command-line interface to `daemon`:
```
usage: daemon [-h] [-c FILE] [-d] [--nrm_log NRM_LOG] [--hwloc HWLOC]
              [--argo_nodeos_config ARGO_NODEOS_CONFIG] [--perf PERF]
              [--argo_perf_wrapper ARGO_PERF_WRAPPER]

optional arguments:
  -h, --help            show this help message and exit
  -c FILE, --configuration FILE
                        Specify a config json-formatted config file to
                        override any of the available CLI options. If an
                        option is actually provided on the command-line, it
                        overrides its corresponding value from the
                        configuration file.
  -d, --print_defaults  Print the default configuration file.
  --nrm_log NRM_LOG     Main log file. Override default with the NRM_LOG.
                        environment variable
  --hwloc HWLOC         Path to the hwloc to use. This path can be relative
                        and makes uses of the $PATH if necessary. Override
                        default with the HWLOC environment variable.
  --argo_nodeos_config ARGO_NODEOS_CONFIG
                        Path to the argo_nodeos_config to use. This path can
                        be relative and makes uses of the $PATH if necessary.
                        Override default with the ARGO_NODEOS_CONFIG
                        environment variable.
  --perf PERF           Path to the linux perf tool to use. This path can be
                        relative and makes uses of the $PATH if necessary.
                        Override default with the PERF environment variable.
  --argo_perf_wrapper ARGO_PERF_WRAPPER
                        Path to the linux perf tool to use. This path can be
                        relative and makes uses of the $PATH if necessary.
                        Override default with the PERFWRAPPER environment
                        variable.
```

Improve process management

See merge request !30

- added correct SIGINT/process ending handling to cmd
- fixed  kill/list containers
- added ZMQ_LINGER 0 to the socket options.

Related to #22

Add a listen command to get access to the event stream of the upstream
pub/sub API.

This patch gives back access from the command line to the power
information of a container, including filtering the event stream to only
have events relevent to this container.

This changes the workflow a little bit for users, but should result in a
cleaner access to profiling data in the future.

Related to #18.

Make it so that the daemon will delete containers when all commands it
is aware of are finished, instead of relying on a single owner that
needs to be tracked.

This simplifies the handling to multiple commands in the same container,
and should not impact the rest.

Move the container start/exit events to the upstream pub/sub event
stream. As these are more of a global event now that we support multiple
commands in the same container, it makes sense to move them to the more
general event stream.

This patch also remove the code in cmd waiting for container start or
exit, making (temporarily) the cmd unable to report power metrics. We
will fix that in a later commit.

This patch fixes complicated issues we had with how a job running
multiple commands in the container might not all wait for the end of the
container: now none of them do.

Add a upstream pub client, to be able to listen to messages coming from
the daemon on the upstream pub/sub channel.

Doesn't support any fancy filter, as that's not used by the daemon so
far.

Ensure that the client that created the container is considered as the
one owning it, with the consequence that if its command exits, the
container is destroyed. Also deals with the race issue we had on the cmd
side.

Current code sends start/exit events when a container is created and
process_start/process_exit when its already there. Instead, have the
container start/exit only care about container stuff, and always sends
the process start/exit events around. That makes the cmd run fsm easier
to work out.

Changes the message format a tiny bit.
Fixes some missing stdout/stderr issues we had before.