README.md 6.56 KB
Newer Older
Valentin Reis's avatar
Valentin Reis committed
1
#### integration testing
Valentin Reis's avatar
Valentin Reis committed
2

Valentin Reis's avatar
Valentin Reis committed
3 4
This repository contains integration tests that validate the argo stack,
leveraging the 'argopkgs' repo
Valentin Reis's avatar
Valentin Reis committed
5

6
The intended usage is to override (some of) the source(s) with WIP
7
version(s), as part of development or continuous integration. This
8 9
gitlab CI snippets shows how to do this on a nix-enabled runner. Mind
that this setup tracks argotest master.
Valentin Reis's avatar
Valentin Reis committed
10

11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
### Usage

-   Get Nix: `curl https://nixos.org/nix/install | sh`

-   Use the 'argotest' nix function to provide an environment, among
    which the `argotk.hs` tool, which runs tests. For example, we can
    run default tests using pinned `argopkgs` sources like so:

``` {.nix}
nix-shell -E '{ argotest ? (builtins.fetchGit {
  url = https://xgitlab.cels.anl.gov/argo/argotest.git;
  ref="master";})
  }: (import argotest {}).test' --run 'argotk.hs helloworld'
```

Alternatively, one can get rid of the `--run` option in order to be
provided with an environment in which all the argo components should be
in the `PATH`, along with the `argotk.hs` tool.

The arguments to the argotest function are defined in the default.nix
file in this repository. They all have default values.

For a more involved example, let's run a test in a custom environment.
Here, we'll run the 'helloworld' test in an environment that uses a
local `nrm` source, the master `libnrm` branch and a specific revision
Valentin Reis's avatar
Valentin Reis committed
36 37
of the `containers` branch. We'l'l use the master `argotest` branch for
that.
38 39 40 41

``` {.nix}
nix-shell -E '{ argotest ? (builtins.fetchGit {
    url = https://xgitlab.cels.anl.gov/argo/argotest.git;
Valentin Reis's avatar
Valentin Reis committed
42
    ref="master";})
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82
}:
(import argotest {
  nrm-src = /path/to/nrm;
  libnrm-src = builtins.fetchGit {
    url = https://xgitlab.cels.anl.gov/argo/libnrm.git;
    ref="master"; };
  containers-src = builtins.fetchGit  {
    url = https://xgitlab.cels.anl.gov/argo/containers.git;
    ref="fancy-branch-name"; 
    rev="commit-revisions-string";
    };
}).test' --run 'argotk.hs helloworld'
```

*WARNING* There are a few things one has to be aware of using this
workflow:

-   Builds may fail if the local source repositories are dirty with old
    build files.

-   Builds may fail if `PWD` is in a `nosuid`-enabled filesystem.

-   Without using the `rev` argument, the `builtins.fetchGit` nix
    command prefetches and buffers its output, with an expiration time
    that ranges ten minutes by default. Use a local checkout if you need
    to modify some of these sources on the fly:

``` {.nix}
nix-shell -E '{ argotest ? /path/to/argotest }:
(import argotest {
  nrm-src = /path/to/nrm;
  # libnrm-src left at default argument value (inherits from argopkgs)
  # containers-src left at default argument value (inherits from argopkgs)
}).test' --run 'argotk.hs helloworld'
```

### Example CI setup

For example, we could setup the CI for the `containers` repo like:

83
``` {.yml}
Valentin Reis's avatar
Valentin Reis committed
84 85 86
integration.test:
  stage: test
  script:
87 88 89 90 91 92
    - nix-shell -E '{ argotest ? (builtins.fetchGit {
          url = https://xgitlab.cels.anl.gov/argo/argotest.git;
          ref="master";})
      }:
      (import argotest { containers-src = ./. ; }).test' 
      --run 'argotk.hs helloworld'
Valentin Reis's avatar
Valentin Reis committed
93 94
  artifacts:
    paths:
95 96 97 98 99 100
      - argotest/_output/cmd_err.log
      - argotest/_output/cmd_out.log
      - argotest/_output/daemon_out.log
      - argotest/_output/daemon_out.log
      - argotest/_output/nrm.log
      - argotest/_output/time.log
Valentin Reis's avatar
Valentin Reis committed
101
    expire_in: 1 week
Valentin Reis's avatar
Valentin Reis committed
102 103 104 105 106 107
  except:
    - /^wip\/.*/
    - /^WIP\/.*/
  tags:
    - integration
```
108

109
### Argotk.hs usage:
110 111

``` {.bash}
112
argotk.hs --help
113 114 115 116 117 118 119 120
```

Output:

``` {.txt}
Usage: argotk.hs COMMAND

Available options:
Valentin Reis's avatar
Valentin Reis committed
121 122 123 124
  COMMAND                  Type of test to run. There are extensive options
                           under each action, but be careful, these do not all
                           have the same defaults. The default values are
                           printed when you call --help on these actions.
125 126 127
  -h,--help                Show this help text

Available commands:
128 129 130
  clean                    Clean sockets, logfiles.
  daemon                   Set up and launch the daemon in synchronous mode,
                           with properly cleaned sockets, logfiles.
Valentin Reis's avatar
Valentin Reis committed
131 132 133
  stack                    Setup stack and run a command in a container.
  grep                     Setup stack and look for a message in the daemon's
                           standard output.
Valentin Reis's avatar
Valentin Reis committed
134
  helloworld               Test 1: Setup stack and check that a hello world app
135
                           sends message back to cmd.
Valentin Reis's avatar
Valentin Reis committed
136 137
  perfwrapper              Test 2: Setup stack and check that argo-perf-wrapper
                           sends at least one progress message up.
138 139 140
```

``` {.bash}
141
argotk.hs helloworld --help
142 143 144
```

``` {.txt}
Valentin Reis's avatar
Valentin Reis committed
145 146 147 148 149 150 151
Usage: argotk.hs helloworld [--application APP]
                            [--container_name ARGO_CONTAINER_UUID]
                            [--output FILE] [--manifest_directory FILE]
                            [--manifest_name FILE] [--cmd_out FILE]
                            [--cmd_err FILE] [--daemon_out FILE]
                            [--daemon_err FILE] [--nrm_log FILE]
                            [--time_file FILE]
Valentin Reis's avatar
Valentin Reis committed
152 153
  Test 1: Setup stack and check that a hello world app sends message back to
  cmd.
154 155

Available options:
Valentin Reis's avatar
Valentin Reis committed
156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
  --application APP        Target application call, sh+path
                           valid (default: "echo helloworld")
  --container_name ARGO_CONTAINER_UUID
                           Container name (default: "testContainer")
  --output FILE            Working directory. (default: FilePath "_output")
  --manifest_directory FILE
                           Manifest lookup
                           directory (default: FilePath "/nix/store/9hbj77178wrp2pv8qs6d063afhdymgk2-manifests")
  --manifest_name FILE     Manifest basename (default: FilePath "basic.json")
  --cmd_out FILE           Output file, application
                           stdout (default: FilePath "cmd_out.log")
  --cmd_err FILE           Output file, application
                           stderr (default: FilePath "cmd_err.log")
  --daemon_out FILE        Output file, daemon
                           stdout (default: FilePath "daemon_out.log")
  --daemon_err FILE        Output file, daemon
                           stderr (default: FilePath "daemon_err.log")
  --nrm_log FILE           Output file, daemon log (default: FilePath "nrm.log")
  --time_file FILE         Output file, application
                           runtime (default: FilePath "time.log")
176 177 178 179
  -h,--help                Show this help text
```

### Hacking
180 181 182 183 184

-   edit `.README.md` in place of README.md.

-   the ./shake.hs build file takes care of a few things for the
    development workflow (readme and completion generation).