GitLab

Make the container Popen pass the working directory from the `nrm
run` call, through the $PWD environment variable.

close_fds only defaults to true on python 3, leading to leaking open
file descriptors every time we fork a process.

BSD3, as discussed.

For some reason this seems to be triggering a leak of a open file
descriptor, even though it should be garbage collected.

As for why it's there...well there was a time when I had hoped of
implementing support for stdin.

Added support for pinning process/task to a core. This is important for
allowing the use of power policies that use contextual information from an
application phase and use it for computing frequency levels for the next phase.
In absence of process/task pinning, the contextual information obtained does
not serve any value as it is not representative of application phase behavior
on a core as the processes and task can migrate during the next phase.

See Issue #20

This commit adds a command-line interface to `daemon`:
```
usage: daemon [-h] [-c FILE] [-d] [--nrm_log NRM_LOG] [--hwloc HWLOC]
              [--argo_nodeos_config ARGO_NODEOS_CONFIG] [--perf PERF]
              [--argo_perf_wrapper ARGO_PERF_WRAPPER]

optional arguments:
  -h, --help            show this help message and exit
  -c FILE, --configuration FILE
                        Specify a config json-formatted config file to
                        override any of the available CLI options. If an
                        option is actually provided on the command-line, it
                        overrides its corresponding value from the
                        configuration file.
  -d, --print_defaults  Print the default configuration file.
  --nrm_log NRM_LOG     Main log file. Override default with the NRM_LOG.
                        environment variable
  --hwloc HWLOC         Path to the hwloc to use. This path can be relative
                        and makes uses of the $PATH if necessary. Override
                        default with the HWLOC environment variable.
  --argo_nodeos_config ARGO_NODEOS_CONFIG
                        Path to the argo_nodeos_config to use. This path can
                        be relative and makes uses of the $PATH if necessary.
                        Override default with the ARGO_NODEOS_CONFIG
                        environment variable.
  --perf PERF           Path to the linux perf tool to use. This path can be
                        relative and makes uses of the $PATH if necessary.
                        Override default with the PERF environment variable.
  --argo_perf_wrapper ARGO_PERF_WRAPPER
                        Path to the linux perf tool to use. This path can be
                        relative and makes uses of the $PATH if necessary.
                        Override default with the PERFWRAPPER environment
                        variable.
```

When invoking 'argo_nodeos_config run', we were passing the job
environment implicitly.  This wasn't very clean and was also causing
problems with variables such as LD_PRELOAD, which were being filtered
out because argo_nodeos_config is suid root.

Use the new argo_nodeos_config --exec feature in development.
Allow us to delegate fork+attach+exec to argo_nodeos_config, and
simplifying the create command as a result.

We use tornado.process to wrap this command, as we want to able to
stream stdout/stderr in the future.

This patch also misuse, the 'pid' field of the container namedtuple to
save the tornado.process.Subprocess object itself, so some functions
need to be adapted.

The logging module allow us to configure logging facilities once per
process using basicConfig, and then to use globally defined, named,
logger objects. This simplifies access to logger objects, their
configuration and remove pointers from all objects.

This patch refactor all the logging calls to use a single 'nrm' logger
object, using those facilities.

This patch refactor the resource management and hwloc code into a
working, albeit very simple scheduling policy. Indeed, the previous code
contained strong assumptions about the output of hwloc matching an Argo
NodeOS configuration used during the previous phase of the project, that
always contained enough CPUs and Mems to perform exclusive scheduling.

The current version is simpler, but should work on more regular systems.
The patch also improves code organization so that introducing more
complex scheduling algorithms will be simpler.

The testing of this code resulted in the discovery of simple bugs in the
daemon children handling code, which should work now.

The Argus (globalos) launcher had prototype code to read a container
manifest, create a container using Judi's code, and map resources using
hwloc.

This patch brings that code, almost intact, into the NRM repo. This code
is quite ugly, and the resource mapping crashes if the kernel
configuration isn't right. But it's still a good starting point, and we
should be able to improve things little by little.

One part in particular needs attention: SIGCHLD handling. We should
think of using ioloop-provided facilities to avoid this mess.

The patch also contains the associated CLI changes.

Note: the messaging format is starting to be difficult to keep in check,
as there's conversions and field checks all over the code. See #3 for
a possible solution.