README.md 8.13 KB
Newer Older
Valentin Reis's avatar
Valentin Reis committed
1
#### integration testing
Valentin Reis's avatar
Valentin Reis committed
2

Valentin Reis's avatar
Valentin Reis committed
3 4
This repository contains integration tests that validate the argo stack,
leveraging the 'argopkgs' repo
Valentin Reis's avatar
Valentin Reis committed
5

6
The intended usage is to override (some of) the source(s) with WIP
7
version(s), as part of development or continuous integration. This
8 9
gitlab CI snippets shows how to do this on a nix-enabled runner. Mind
that this setup tracks argotest master.
Valentin Reis's avatar
Valentin Reis committed
10

Valentin Reis's avatar
Valentin Reis committed
11
### Usage (tl;dr, I already have nix on my machine.)
12

Valentin Reis's avatar
Valentin Reis committed
13 14 15
``` {.bash}
nix-shell -E '
(import( builtins.fetchGit {
16
  url = https://xgitlab.cels.anl.gov/argo/argotest.git;
Valentin Reis's avatar
Valentin Reis committed
17 18 19 20 21 22
  ref="master";
}) {
  #nrm-src = /path/to/nrm
  #libnrm-src = /path/to/nrm
  #containers-src = /path/to/nrm
  }).test' --run "argotk.hs helloworld"
23 24
```

Valentin Reis's avatar
Valentin Reis committed
25 26 27 28 29 30 31
### Usage (in three parts)

-   \[*1*\] Get Nix: `curl https://nixos.org/nix/install | sh`

-   \[*2*\] Use the ~test~ attribute of the argotest' nix attribute set
    to enter a test environment. For example, we can run default tests
    on the "argopkgs-pinned" version of argo components using:
32

Valentin Reis's avatar
Valentin Reis committed
33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
``` {.bash}
nix-shell -E '
let
  argotest-src =
    builtins.fetchGit {
      url = https://xgitlab.cels.anl.gov/argo/argotest.git;
      ref="master";
    };
  argotest = import argotest-src {};
in
  argotest.test'
```

This environment has all the necessary Argo components in its PATH. The
argotest function has various arguments, defined in the default.nix file
at the rooto of this repository. They all have default values. For a
more involved example, let's get a custom test environment.
50

Valentin Reis's avatar
Valentin Reis committed
51 52 53
Here, we'll use an environment that uses a local `nrm` source, the
master `libnrm` branch and a specific revision of the `containers`
branch. We'l'l use the master `argotest` branch for that:
54 55 56 57

``` {.nix}
nix-shell -E '{ argotest ? (builtins.fetchGit {
    url = https://xgitlab.cels.anl.gov/argo/argotest.git;
Valentin Reis's avatar
Valentin Reis committed
58
    ref="master";})
59 60 61 62 63 64 65 66 67 68 69 70 71 72
}:
(import argotest {
  nrm-src = /path/to/nrm;
  libnrm-src = builtins.fetchGit {
    url = https://xgitlab.cels.anl.gov/argo/libnrm.git;
    ref="master"; };
  containers-src = builtins.fetchGit  {
    url = https://xgitlab.cels.anl.gov/argo/containers.git;
    ref="fancy-branch-name"; 
    rev="commit-revisions-string";
    };
}).test' --run 'argotk.hs helloworld'
```

Valentin Reis's avatar
Valentin Reis committed
73 74
-   \[3\] The `test`environment contains the `argotk.hs` tool, which
    runs various operations on the argo stack:
75

Valentin Reis's avatar
Valentin Reis committed
76 77
Commands list:

78
``` {.bash}
79
argotk.hs --help
80 81 82 83 84 85 86 87
```

Output:

``` {.txt}
Usage: argotk.hs COMMAND

Available options:
Valentin Reis's avatar
Valentin Reis committed
88 89 90 91
  COMMAND                  Type of test to run. There are extensive options
                           under each action, but be careful, these do not all
                           have the same defaults. The default values are
                           printed when you call --help on these actions.
92 93 94
  -h,--help                Show this help text

Available commands:
95
  clean                    Clean sockets, logfiles.
Valentin Reis's avatar
Valentin Reis committed
96
  daemon-only              Set up and launch the daemon in synchronous mode,
97
                           with properly cleaned sockets, logfiles.
Valentin Reis's avatar
Valentin Reis committed
98
  full-stack               Setup stack and run a command in a container.
Valentin Reis's avatar
Valentin Reis committed
99
  helloworld               Test 1: Setup stack and check that a hello world app
100
                           sends message back to cmd.
Valentin Reis's avatar
Valentin Reis committed
101 102
  perfwrapper              Test 2: Setup stack and check that argo-perf-wrapper
                           sends at least one progress message up.
103 104
```

Valentin Reis's avatar
Valentin Reis committed
105 106
Detailed help:

107
``` {.bash}
108
argotk.hs helloworld --help
109 110
```

Valentin Reis's avatar
Valentin Reis committed
111 112
Output:

113
``` {.txt}
Valentin Reis's avatar
Valentin Reis committed
114 115 116 117 118 119
Usage: argotk.hs helloworld [--application APP]
                            [--container_name ARGO_CONTAINER_UUID]
                            [--output FILE] [--manifest_directory FILE]
                            [--manifest_name FILE] [--cmd_out FILE]
                            [--cmd_err FILE] [--daemon_out FILE]
                            [--daemon_err FILE] [--nrm_log FILE]
Valentin Reis's avatar
Valentin Reis committed
120
                            [--message_daemon_stdout STRING]
Valentin Reis's avatar
Valentin Reis committed
121 122 123
                            [--message_daemon_stderr STRING]
                            [--message_cmd_stdout STRING]
                            [--message_cmd_stderr STRING]
Valentin Reis's avatar
Valentin Reis committed
124 125
  Test 1: Setup stack and check that a hello world app sends message back to
  cmd.
126 127

Available options:
Valentin Reis's avatar
Valentin Reis committed
128 129
  --application APP        Target application executable name. PATH is
                           inherited. (default: "echo")
Valentin Reis's avatar
Valentin Reis committed
130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145
  --container_name ARGO_CONTAINER_UUID
                           Container name (default: "testContainer")
  --output FILE            Working directory. (default: FilePath "_output")
  --manifest_directory FILE
                           Manifest lookup
                           directory (default: FilePath "/nix/store/9hbj77178wrp2pv8qs6d063afhdymgk2-manifests")
  --manifest_name FILE     Manifest basename (default: FilePath "basic.json")
  --cmd_out FILE           Output file, application
                           stdout (default: FilePath "cmd_out.log")
  --cmd_err FILE           Output file, application
                           stderr (default: FilePath "cmd_err.log")
  --daemon_out FILE        Output file, daemon
                           stdout (default: FilePath "daemon_out.log")
  --daemon_err FILE        Output file, daemon
                           stderr (default: FilePath "daemon_err.log")
  --nrm_log FILE           Output file, daemon log (default: FilePath "nrm.log")
Valentin Reis's avatar
Valentin Reis committed
146 147 148 149 150
  --message_daemon_stdout STRING
                           The appearance of this character string in the daemon
                           stdout will be monitored during execution and the
                           stack will be killed when observing it, returning a
                           successful exit code.
Valentin Reis's avatar
Valentin Reis committed
151
  --message_daemon_stderr STRING
Valentin Reis's avatar
Valentin Reis committed
152 153 154 155
                           The appearance of this character string in the daemon
                           stderr will be monitored during execution and the
                           stack will be killed when observing it, returning a
                           successful exit code.
Valentin Reis's avatar
Valentin Reis committed
156
  --message_cmd_stdout STRING
Valentin Reis's avatar
Valentin Reis committed
157 158 159 160
                           The appearance of this character string in the cmd
                           stdout will be monitored during execution and the
                           stack will be killed when observing it, returning a
                           successful exit code. (default: "Hello-Moto")
Valentin Reis's avatar
Valentin Reis committed
161
  --message_cmd_stderr STRING
Valentin Reis's avatar
Valentin Reis committed
162 163 164 165
                           The appearance of this character string in the cmd
                           stderr will be monitored during execution and the
                           stack will be killed when observing it, returning a
                           successful exit code. (default: "Hello-Moto")
166 167 168
  -h,--help                Show this help text
```

Valentin Reis's avatar
Valentin Reis committed
169
#### Misc
Valentin Reis's avatar
Valentin Reis committed
170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224

Alternatively, one can use the `--run` option to run a test directly:

``` {.bash}
nix-shell -E '
(import( builtins.fetchGit {
  url = https://xgitlab.cels.anl.gov/argo/argotest.git;
  ref="master";
}) {}).test' --run "argotk.hs helloworld"
```

#### WARNINGS

There are a few things one has to be aware of using this workflow:

-   Builds may fail if the local source repositories are dirty with old
    build files.

-   Builds may fail if `PWD` is in a `nosuid`-enabled filesystem.

-   Without using the `rev` argument, the `builtins.fetchGit` nix
    command prefetches and buffers its output, with an expiration time
    that ranges ten minutes by default. Use a local checkout if you need
    to modify some of these sources on the fly:

### Example CI setup

For example, we could setup the CI for the `containers` repo like:

``` {.yml}
integration.test:
  stage: test
  script:
    - nix-shell -E '{ argotest ? (builtins.fetchGit {
          url = https://xgitlab.cels.anl.gov/argo/argotest.git;
          ref="master";})
      }:
      (import argotest { containers-src = ./. ; }).test' 
      --run 'argotk.hs helloworld'
  artifacts:
    paths:
      - argotest/_output/cmd_err.log
      - argotest/_output/cmd_out.log
      - argotest/_output/daemon_out.log
      - argotest/_output/daemon_out.log
      - argotest/_output/nrm.log
      - argotest/_output/time.log
    expire_in: 1 week
  except:
    - /^wip\/.*/
    - /^WIP\/.*/
  tags:
    - integration
```

225
### Hacking
226 227 228 229 230

-   edit `.README.md` in place of README.md.

-   the ./shake.hs build file takes care of a few things for the
    development workflow (readme and completion generation).