README.md 9.84 KB
Newer Older
Valentin Reis's avatar
Valentin Reis committed
1
#### integration testing
Valentin Reis's avatar
Valentin Reis committed
2

Valentin Reis's avatar
Valentin Reis committed
3 4 5 6
Integration tests that validate the argo stack, leveraging the
'argopkgs' repository. The intended usage is to override (some of) the
source(s) with WIP version(s), as part of development or continuous
integration.
Valentin Reis's avatar
Valentin Reis committed
7

Valentin Reis's avatar
Valentin Reis committed
8
### Usage (tl;dr, I already have nix on my machine.)
9

Valentin Reis's avatar
Valentin Reis committed
10 11 12
``` {.bash}
nix-shell -E '
(import( builtins.fetchGit {
13
  url = https://xgitlab.cels.anl.gov/argo/argotest.git;
Valentin Reis's avatar
Valentin Reis committed
14 15 16 17 18
  ref="master";
}) {
  #nrm-src = /path/to/nrm
  #libnrm-src = /path/to/nrm
  #containers-src = /path/to/nrm
Valentin Reis's avatar
Valentin Reis committed
19
  }).test' --run "argotk.hs TestHello"
20 21
```

Valentin Reis's avatar
Valentin Reis committed
22 23
### Usage (in three parts)

Valentin Reis's avatar
Valentin Reis committed
24
-   \[**1**\] Get Nix: `curl https://nixos.org/nix/install | sh`
Valentin Reis's avatar
Valentin Reis committed
25

Valentin Reis's avatar
Valentin Reis committed
26 27 28
-   \[**2**\] Use the `test` attribute of the argotest' nix attribute
    set to enter a test environment. For example, we can run default
    tests on the "argopkgs-pinned" version of argo components using:
29

Valentin Reis's avatar
Valentin Reis committed
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
``` {.bash}
nix-shell -E '
let
  argotest-src =
    builtins.fetchGit {
      url = https://xgitlab.cels.anl.gov/argo/argotest.git;
      ref="master";
    };
  argotest = import argotest-src {};
in
  argotest.test'
```

This environment has all the necessary Argo components in its PATH. The
argotest function has various arguments, defined in the default.nix file
at the rooto of this repository. They all have default values. For a
more involved example, let's get a custom test environment.
47

Valentin Reis's avatar
Valentin Reis committed
48 49 50
Here, we'll use an environment that uses a local `nrm` source, the
master `libnrm` branch and a specific revision of the `containers`
branch. We'l'l use the master `argotest` branch for that:
51 52 53 54

``` {.nix}
nix-shell -E '{ argotest ? (builtins.fetchGit {
    url = https://xgitlab.cels.anl.gov/argo/argotest.git;
Valentin Reis's avatar
Valentin Reis committed
55
    ref="master";})
56 57 58 59 60 61 62 63 64 65 66
}:
(import argotest {
  nrm-src = /path/to/nrm;
  libnrm-src = builtins.fetchGit {
    url = https://xgitlab.cels.anl.gov/argo/libnrm.git;
    ref="master"; };
  containers-src = builtins.fetchGit  {
    url = https://xgitlab.cels.anl.gov/argo/containers.git;
    ref="fancy-branch-name"; 
    rev="commit-revisions-string";
    };
Valentin Reis's avatar
Valentin Reis committed
67
}).test' --run 'argotk.hs TestHello'
68 69
```

Valentin Reis's avatar
Valentin Reis committed
70
-   \[**3**\] The `test`environment contains the `argotk.hs` tool, which
Valentin Reis's avatar
Valentin Reis committed
71
    runs various operations on the argo stack:
72

Valentin Reis's avatar
Valentin Reis committed
73 74
Commands list:

75
``` {.bash}
76
argotk.hs --help
77 78 79 80 81 82 83 84
```

Output:

``` {.txt}
Usage: argotk.hs COMMAND

Available options:
Valentin Reis's avatar
Valentin Reis committed
85 86 87 88
  COMMAND                  Type of test to run. There are extensive options
                           under each action, but be careful, these do not all
                           have the same defaults. The default values are
                           printed when you call --help on these actions.
89 90 91
  -h,--help                Show this help text

Available commands:
92
  clean                    Clean sockets, logfiles.
Valentin Reis's avatar
Valentin Reis committed
93
  daemon-only              Set up and launch the daemon in synchronous mode,
94
                           with properly cleaned sockets, logfiles.
Valentin Reis's avatar
Valentin Reis committed
95
  full-stack               Setup stack and run a command in a container.
Valentin Reis's avatar
Valentin Reis committed
96
  TestHello                test1: Setup stack and check that a hello world app
97
                           sends message back to cmd.
Valentin Reis's avatar
Valentin Reis committed
98 99 100 101 102 103 104
  TestListen               test2: Setup stack and check that argo-perf-wrapper
                           sends at least one message to the daemon.
  TestPerfwrapper          test3: Setup stack and check that argo-perf-wrapper
                           sends at least one *performance* message to cmd
                           listen through the daemon.
  TestPower                test4: Setup stack and check that argo-perf-wrapper
                           sends at least one *power* message to cmd listen
Valentin Reis's avatar
Valentin Reis committed
105
                           through the daemon.
Valentin Reis's avatar
Valentin Reis committed
106
  tests                    Run all tests
107 108
```

Valentin Reis's avatar
Valentin Reis committed
109 110
Detailed help:

111
``` {.bash}
Valentin Reis's avatar
Valentin Reis committed
112
argotk.hs TestHello --help
113 114
```

Valentin Reis's avatar
Valentin Reis committed
115 116
Output:

117
``` {.txt}
Valentin Reis's avatar
Valentin Reis committed
118 119 120 121 122 123 124 125 126 127 128 129 130 131
Usage: argotk.hs TestHello [--application APP]
                           [--container_name ARGO_CONTAINER_UUID]
                           [--output_dir DIR] [--manifest_directory DIR]
                           [--manifest_name FILENAME] [--cmd_run_out FILENAME]
                           [--cmd_run_err FILENAME] [--cmd_listen_out FILENAME]
                           [--cmd_listen_err FILENAME] [--daemon_out FILENAME]
                           [--daemon_err FILENAME] [--nrm_log FILENAME]
                           [--message_daemon_stdout STRING]
                           [--message_daemon_stderr STRING]
                           [--message_cmd_run_stdout STRING]
                           [--message_cmd_run_stderr STRING]
                           [--message_cmd_listen_stdout STRING]
                           [--message_cmd_listen_stderr STRING]
  test1: Setup stack and check that a hello world app sends message back to cmd.
132 133

Available options:
Valentin Reis's avatar
Valentin Reis committed
134 135
  --application APP        Target application executable name. PATH is
                           inherited. (default: "echo")
Valentin Reis's avatar
Valentin Reis committed
136 137
  --container_name ARGO_CONTAINER_UUID
                           Container name (default: "testContainer")
138 139
  --output_dir DIR         Working directory. (default: FilePath "_output")
  --manifest_directory DIR Manifest lookup
Valentin Reis's avatar
Valentin Reis committed
140
                           directory (default: FilePath "/nix/store/9hbj77178wrp2pv8qs6d063afhdymgk2-manifests")
141 142
  --manifest_name FILENAME Manifest file basename (relative to
                           --manifest_directory) (default: FilePath "basic.json")
Valentin Reis's avatar
Valentin Reis committed
143 144 145 146 147 148 149 150 151 152
  --cmd_run_out FILENAME   Output file (relative to --output_dir), "cmd run"
                           stdout (default: FilePath "cmd_run_out.log")
  --cmd_run_err FILENAME   Output file (relative to --output_dir), "cmd run"
                           stderr (default: FilePath "cmd_run_err.log")
  --cmd_listen_out FILENAME
                           Output file (relative to --output_dir), "cmd listen"
                           stdout (default: FilePath "cmd_listen_out.log")
  --cmd_listen_err FILENAME
                           Output file (relative to --output_dir), "cmd listen"
                           stderr (default: FilePath "cmd_listen_err.log")
153
  --daemon_out FILENAME    Output file (relative to --output_dir), daemon
Valentin Reis's avatar
Valentin Reis committed
154
                           stdout (default: FilePath "daemon_out.log")
155
  --daemon_err FILENAME    Output file (relative to --output_dir), daemon
Valentin Reis's avatar
Valentin Reis committed
156
                           stderr (default: FilePath "daemon_err.log")
157 158
  --nrm_log FILENAME       Output file (relative to --output_dir), daemon
                           log (default: FilePath "nrm.log")
Valentin Reis's avatar
Valentin Reis committed
159 160
  --message_daemon_stdout STRING
                           The appearance of this character string in the daemon
161 162 163
                           stdout will be monitored during execution. When
                           observed, the stack will be killed and a return code
                           of 0 will be returned.
Valentin Reis's avatar
Valentin Reis committed
164
  --message_daemon_stderr STRING
Valentin Reis's avatar
Valentin Reis committed
165
                           The appearance of this character string in the daemon
166 167 168
                           stdout will be monitored during execution. When
                           observed, the stack will be killed and a return code
                           of 0 will be returned.
Valentin Reis's avatar
Valentin Reis committed
169 170 171
  --message_cmd_run_stdout STRING
                           The appearance of this character string in the cmd
                           run stdout will be monitored during execution. When
172
                           observed, the stack will be killed and a return code
Valentin Reis's avatar
Valentin Reis committed
173 174
                           of 0 will be
                           returned. (default: "someComplicatedMessage")
Valentin Reis's avatar
Valentin Reis committed
175 176 177
  --message_cmd_run_stderr STRING
                           The appearance of this character string in the cmd
                           run stdout will be monitored during execution. When
178
                           observed, the stack will be killed and a return code
Valentin Reis's avatar
Valentin Reis committed
179 180
                           of 0 will be
                           returned. (default: "someComplicatedMessage")
Valentin Reis's avatar
Valentin Reis committed
181 182 183 184 185 186 187 188 189 190
  --message_cmd_listen_stdout STRING
                           The appearance of this character string in the cmd
                           listen stdout will be monitored during execution.
                           When observed, the stack will be killed and a return
                           code of 0 will be returned.
  --message_cmd_listen_stderr STRING
                           The appearance of this character string in the cmd
                           listen stdout will be monitored during execution.
                           When observed, the stack will be killed and a return
                           code of 0 will be returned.
191 192 193
  -h,--help                Show this help text
```

Valentin Reis's avatar
Valentin Reis committed
194
#### Misc
Valentin Reis's avatar
Valentin Reis committed
195 196 197 198 199 200 201 202

Alternatively, one can use the `--run` option to run a test directly:

``` {.bash}
nix-shell -E '
(import( builtins.fetchGit {
  url = https://xgitlab.cels.anl.gov/argo/argotest.git;
  ref="master";
Valentin Reis's avatar
Valentin Reis committed
203
}) {}).test' --run "argotk.hs TestHello"
Valentin Reis's avatar
Valentin Reis committed
204 205 206 207 208 209 210 211 212 213 214 215
```

#### WARNINGS

There are a few things one has to be aware of using this workflow:

-   Builds may fail if the local source repositories are dirty with old
    build files.

-   Without using the `rev` argument, the `builtins.fetchGit` nix
    command prefetches and buffers its output, with an expiration time
    that ranges ten minutes by default. Use a local checkout if you need
Valentin Reis's avatar
Valentin Reis committed
216
    to modify some of these sources on the fly.
Valentin Reis's avatar
Valentin Reis committed
217 218 219 220 221 222 223 224 225 226 227

### Example CI setup

``` {.yml}
integration.test:
  stage: test
  script:
    - nix-shell -E '{ argotest ? (builtins.fetchGit {
          url = https://xgitlab.cels.anl.gov/argo/argotest.git;
          ref="master";})
      }:
Valentin Reis's avatar
Valentin Reis committed
228
      (import argotest { containers-src = ./. ; }).test' \
Valentin Reis's avatar
Valentin Reis committed
229
      --run 'argotk.hs TestHello'
Valentin Reis's avatar
Valentin Reis committed
230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245
  artifacts:
    paths:
      - argotest/_output/cmd_err.log
      - argotest/_output/cmd_out.log
      - argotest/_output/daemon_out.log
      - argotest/_output/daemon_out.log
      - argotest/_output/nrm.log
      - argotest/_output/time.log
    expire_in: 1 week
  except:
    - /^wip\/.*/
    - /^WIP\/.*/
  tags:
    - integration
```

246
### Hacking
247 248 249 250 251

-   edit `.README.md` in place of README.md.

-   the ./shake.hs build file takes care of a few things for the
    development workflow (readme and completion generation).