1. 06 Feb, 2019 5 commits
    • Swann Perarnau's avatar
      [feature] add a dummy container runtime · 8c547615
      Swann Perarnau authored
      Add a container runtime that doesn't create containers, but can still
      launch commands.
      
      That should make it possible, once we figure out how to change it in the
      configuration, to actually run the NRM without a setuid binary.
      8c547615
    • Swann Perarnau's avatar
      [refactor] add an abstraction for container runtimes · c8ca4fa5
      Swann Perarnau authored
      Extract the container runtime interface from the container manager, and
      use a hierarchy of classes to enforce a runtime interface that makes
      sense.
      
      This will allow us to create alternative runtime implementations without
      major changes to the container manager code.
      c8ca4fa5
    • Swann Perarnau's avatar
      [log] adding binding output to create log · 1fa07af3
      Swann Perarnau authored
      Just to make it easy to debug binding issues.
      1fa07af3
    • Swann Perarnau's avatar
      [fix] restore the use of resources tuple · dca6debb
      Swann Perarnau authored
      !27 replaced the resource field of the `container` named tuple by a
      dictionary. Restore the old type, a `resources` named tuple and make
      sure to propagate this across all the code.
      dca6debb
    • Swann Perarnau's avatar
      [refactor] extract out basic container management · eb7c6380
      Swann Perarnau authored
      Refactor the container creation code to isolate the building/retrieving
      of a container namedtuple out of the core container creation code.
      
      This simplifies quite a bit the different branches of this code, and
      makes the core create method almost entirely dedicated to just launching
      a command.
      eb7c6380
  2. 05 Feb, 2019 4 commits
  3. 04 Feb, 2019 1 commit
  4. 21 Jan, 2019 3 commits
    • Sridutt Bhalachandra's avatar
      [fix] Aggregative downstream & new msg layer · f3c53106
      Sridutt Bhalachandra authored
      Made necesseary fixes required to make the aggregative downstream api
      integration to work with the new downstream messaging layer.
      
      Also,fixed the case where daemon crashed when an application message
      (from libnrm using pmpi) was received after container was killed
      
      run_policy on all containers removed as the controller no longer has
      application manager info
      
      Any other refactoring and fixes required (check merge request
      discussion)
      
      See Issues #13, #20 and Merge !41
      f3c53106
    • Sridutt Bhalachandra's avatar
      [feature] Support for Process/Task pinning · 402a2524
      Sridutt Bhalachandra authored
      Added support for pinning process/task to a core. This is important for
      allowing the use of power policies that use contextual information from an
      application phase and use it for computing frequency levels for the next phase.
      In absence of process/task pinning, the contextual information obtained does
      not serve any value as it is not representative of application phase behavior
      on a core as the processes and task can migrate during the next phase.
      
      See Issue #20
      402a2524
    • Sridutt Bhalachandra's avatar
      [fix] Multi-node support and msg layer interaction · 33316192
      Sridutt Bhalachandra authored
      Fixed the interaction of the multi-node support feature (#17) with the new
      messaging layer feature. Also, added any other fixes required to make the
      libnrm work with the Aggregative downstream API
      33316192
  5. 18 Dec, 2018 1 commit
  6. 12 Dec, 2018 2 commits
    • Valentin Reis's avatar
      [fix] Fixes the process kill call. · ec503ffa
      Valentin Reis authored
      Fixing a bug introduced by the 'progress-report' branch in a recent
      previous commit. The process object is the result of a tornado spawn, so
      the call has to be slightly different than what was there.
      ec503ffa
    • Valentin Reis's avatar
      [Feature] Adds configuration management and environment variables · 25443c64
      Valentin Reis authored
      This commit adds a command-line interface to `daemon`:
      ```
      usage: daemon [-h] [-c FILE] [-d] [--nrm_log NRM_LOG] [--hwloc HWLOC]
                    [--argo_nodeos_config ARGO_NODEOS_CONFIG] [--perf PERF]
                    [--argo_perf_wrapper ARGO_PERF_WRAPPER]
      
      optional arguments:
        -h, --help            show this help message and exit
        -c FILE, --configuration FILE
                              Specify a config json-formatted config file to
                              override any of the available CLI options. If an
                              option is actually provided on the command-line, it
                              overrides its corresponding value from the
                              configuration file.
        -d, --print_defaults  Print the default configuration file.
        --nrm_log NRM_LOG     Main log file. Override default with the NRM_LOG.
                              environment variable
        --hwloc HWLOC         Path to the hwloc to use. This path can be relative
                              and makes uses of the $PATH if necessary. Override
                              default with the HWLOC environment variable.
        --argo_nodeos_config ARGO_NODEOS_CONFIG
                              Path to the argo_nodeos_config to use. This path can
                              be relative and makes uses of the $PATH if necessary.
                              Override default with the ARGO_NODEOS_CONFIG
                              environment variable.
        --perf PERF           Path to the linux perf tool to use. This path can be
                              relative and makes uses of the $PATH if necessary.
                              Override default with the PERF environment variable.
        --argo_perf_wrapper ARGO_PERF_WRAPPER
                              Path to the linux perf tool to use. This path can be
                              relative and makes uses of the $PATH if necessary.
                              Override default with the PERFWRAPPER environment
                              variable.
      ```
      25443c64
  7. 10 Dec, 2018 1 commit
    • Valentin Reis's avatar
      Small work session with Swann: · 5b550e0b
      Valentin Reis authored
      - added correct SIGINT/process ending handling to cmd
      - fixed  kill/list containers
      - added ZMQ_LINGER 0 to the socket options.
      5b550e0b
  8. 21 Oct, 2018 1 commit
    • Swann Perarnau's avatar
      [refactor] replace upstream comms with msg layer · 0b0ab966
      Swann Perarnau authored
      Replace the fragile upstream communications with the new messaging
      layer, improving the stability and performance of this API.
      
      NOTE: this breaks previous clients
      NOTE: this patch is missing client tracking, to handle children signals.
      0b0ab966
  9. 17 Oct, 2018 1 commit
    • Sridutt Bhalachandra's avatar
      [Feature] Multi- node and process support · 5a41baba
      Sridutt Bhalachandra authored
      Added multi- node and process support that will allow launching of
      multiple processes within a container. This is important for enabling
      use of NRM with MPI applications with multiple processes in a container
      and thus enabling multi-node executions
      
      See Issue #17
      5a41baba
  10. 15 Aug, 2018 1 commit
  11. 14 Aug, 2018 1 commit
  12. 10 Aug, 2018 3 commits
  13. 17 Jul, 2018 1 commit
  14. 21 Dec, 2017 1 commit
  15. 19 Dec, 2017 3 commits
  16. 15 Dec, 2017 1 commit
    • Swann Perarnau's avatar
      [refactor] Only track container inside the CM · f2bc8b80
      Swann Perarnau authored
      The daemon code was maintaining its own container tracker using pids,
      instead of using the one in the container manager. This patch removes
      this additional tracking, and let the daemon side deal with an actual
      namedtuple.
      f2bc8b80
  17. 14 Dec, 2017 5 commits
    • Swann Perarnau's avatar
      [refactor] Fix container namedtuple · 9afe59c7
      Swann Perarnau authored
      This patch propagates the process object into the container namedtuple,
      fix a couple of bad function calls and adapt the run command handler to
      use that process object instead of just the pid of it.
      9afe59c7
    • Swann Perarnau's avatar
      [feature] Use argo_nodeos_config --exec · edeb413b
      Swann Perarnau authored
      Use the new argo_nodeos_config --exec feature in development.
      Allow us to delegate fork+attach+exec to argo_nodeos_config, and
      simplifying the create command as a result.
      
      We use tornado.process to wrap this command, as we want to able to
      stream stdout/stderr in the future.
      
      This patch also misuse, the 'pid' field of the container namedtuple to
      save the tornado.process.Subprocess object itself, so some functions
      need to be adapted.
      edeb413b
    • Swann Perarnau's avatar
      [fix] Fix missing logging changes · ed33ef6d
      Swann Perarnau authored
      The logging improvement patch missed a few calls.
      ed33ef6d
    • Swann Perarnau's avatar
      [refactor] Use globally configured logger · b66c88ec
      Swann Perarnau authored
      The logging module allow us to configure logging facilities once per
      process using basicConfig, and then to use globally defined, named,
      logger objects. This simplifies access to logger objects, their
      configuration and remove pointers from all objects.
      
      This patch refactor all the logging calls to use a single 'nrm' logger
      object, using those facilities.
      b66c88ec
    • Swann Perarnau's avatar
      [refactor] Allow updates in resource tracking · d5f88a14
      Swann Perarnau authored
      Implement an update allocation function to be able to update resource
      tracking when containers are created and deleted.
      
      The commit should make it easier to improve the resource manager later
      on.
      d5f88a14
  18. 13 Dec, 2017 3 commits
    • Swann Perarnau's avatar
      [feature] Add kill command · 63c2dea8
      Swann Perarnau authored
      This patch adds a command to kill the parent process of a container
      based on the container uuid, triggering the death of the container.
      
      The os.kill command interacts pretty badly with the custom built
      children handling, causing us to catch unwanted exceptions in an effort
      to keep the code running. The waitpid code was also missing a bit about
      catching children exiting because of signals, so we fixed that.
      
      At this point, two things should be paid attention to:
        - we don't distinguish properly between a container and a command.
        This will probably cause issues later, as it should be possible to
        launch multiple programs in the same container, and for partitions to
        survive the death of the parent process.
        - the message format is growing more complex, but without any
        component having strong ownership over it. This will probably cause
        stability issues in the long term, as the format complexifies and we
        lose track of the fields expected from everyone.
      63c2dea8
    • Swann Perarnau's avatar
      [feature] Add command to list containers · 2f470afb
      Swann Perarnau authored
      This patch adds a very simple command to list the containers currently
      known by the NRM. There's no history or state tracking on the NRM, so
      the code is pretty simple.
      
      We expect that some of the container tracking doesn't need to be sent
      for such a command, so the listing also filters some of the fields.
      
      This patch also adds an 'event' field to container messages, as it would
      probably be needed further for other kind of operations.
      2f470afb
    • Swann Perarnau's avatar
      [feature] Implement simple RM for containers · 1c4645cc
      Swann Perarnau authored
      This patch refactor the resource management and hwloc code into a
      working, albeit very simple scheduling policy. Indeed, the previous code
      contained strong assumptions about the output of hwloc matching an Argo
      NodeOS configuration used during the previous phase of the project, that
      always contained enough CPUs and Mems to perform exclusive scheduling.
      
      The current version is simpler, but should work on more regular systems.
      The patch also improves code organization so that introducing more
      complex scheduling algorithms will be simpler.
      
      The testing of this code resulted in the discovery of simple bugs in the
      daemon children handling code, which should work now.
      1c4645cc
  19. 11 Dec, 2017 1 commit
    • Swann Perarnau's avatar
      [feature] Pull the Argus code into the NRM · 92290b22
      Swann Perarnau authored
      The Argus (globalos) launcher had prototype code to read a container
      manifest, create a container using Judi's code, and map resources using
      hwloc.
      
      This patch brings that code, almost intact, into the NRM repo. This code
      is quite ugly, and the resource mapping crashes if the kernel
      configuration isn't right. But it's still a good starting point, and we
      should be able to improve things little by little.
      
      One part in particular needs attention: SIGCHLD handling. We should
      think of using ioloop-provided facilities to avoid this mess.
      
      The patch also contains the associated CLI changes.
      
      Note: the messaging format is starting to be difficult to keep in check,
      as there's conversions and field checks all over the code. See #3 for
      a possible solution.
      92290b22