Commit 28801c43 authored by Valentin Reis's avatar Valentin Reis

Various updates, including development workflow and readme.

parent 3f6c8067
Pipeline #4712 passed with stage
in 9 seconds
...@@ -7,25 +7,95 @@ their master branch counterparts. see file `default.nix` for details. ...@@ -7,25 +7,95 @@ their master branch counterparts. see file `default.nix` for details.
The intended usage is to override (some of) the source(s) with WIP The intended usage is to override (some of) the source(s) with WIP
version(s), as part of development or continuous integration. This version(s), as part of development or continuous integration. This
gitlab CI snippets shows how to do this on a nix-enabled runner. Mind that gitlab CI snippets shows how to do this on a nix-enabled runner. Mind
this setup tracks argotest master. that this setup tracks argotest master.
```{.yml} ### Usage
- Get Nix: `curl https://nixos.org/nix/install | sh`
- Use the 'argotest' nix function to provide an environment, among which the
`argotk.hs` tool, which runs tests. For example, we can run default tests
using pinned `argopkgs` sources like so:
```nix
nix-shell -E '{ argotest ? (builtins.fetchGit {
url = https://xgitlab.cels.anl.gov/argo/argotest.git;
ref="master";})
}: (import argotest {}).test' --run 'argotk.hs helloworld'
```
Alternatively, one can get rid of the `--run` option in order to be provided
with an environment in which all the argo components should be in the `PATH`,
along with the `argotk.hs` tool.
The arguments to the argotest function are defined in the default.nix file in
this repository. They all have default values.
For a more involved example, let's run a test in a custom environment. Here,
we'll run the 'helloworld' test in an environment that uses a local `nrm`
source, the master `libnrm` branch and a specific revision of the `containers`
branch:
```nix
nix-shell -E '{ argotest ? (builtins.fetchGit {
url = https://xgitlab.cels.anl.gov/argo/argotest.git;
ref="argo-nodeos-config-suid-dir";})
}:
(import argotest {
nrm-src = /path/to/nrm;
libnrm-src = builtins.fetchGit {
url = https://xgitlab.cels.anl.gov/argo/libnrm.git;
ref="master"; };
containers-src = builtins.fetchGit {
url = https://xgitlab.cels.anl.gov/argo/containers.git;
ref="fancy-branch-name";
rev="commit-revisions-string";
};
}).test' --run 'argotk.hs helloworld'
```
*WARNING* There are a few things one has to be aware of using this workflow:
- Builds may fail if the local source repositories are dirty with old build files.
- Builds may fail if `PWD` is in a `nosuid`-enabled filesystem.
- Without using the `rev` argument, the `builtins.fetchGit` nix command
prefetches and buffers its output, with an expiration time that ranges ten
minutes by default. Use a local checkout if you need to modify some of these
sources on the fly:
```nix
nix-shell -E '{ argotest ? /path/to/argotest }:
(import argotest {
nrm-src = /path/to/nrm;
# libnrm-src left at default argument value (inherits from argopkgs)
# containers-src left at default argument value (inherits from argopkgs)
}).test' --run 'argotk.hs helloworld'
```
### Example CI setup
For example, we could setup the CI for the `containers` repo like:
``` {.yml}
integration.test: integration.test:
stage: test stage: test
script: script:
- BUILD=$PWD - nix-shell -E '{ argotest ? (builtins.fetchGit {
- git clone https://xgitlab.cels.anl.gov/argo/argotest.git url = https://xgitlab.cels.anl.gov/argo/argotest.git;
- cd argotest ref="master";})
- nix-shell -A test --run "./argotk.hs helloworld" --arg nrm-src ../. }:
(import argotest { containers-src = ./. ; }).test'
--run 'argotk.hs helloworld'
artifacts: artifacts:
paths: paths:
- argotest/cmd_err.log - argotest/_output/cmd_err.log
- argotest/cmd_out.log - argotest/_output/cmd_out.log
- argotest/daemon_out.log - argotest/_output/daemon_out.log
- argotest/daemon_out.log - argotest/_output/daemon_out.log
- argotest/nrm.log - argotest/_output/nrm.log
- argotest/time.log - argotest/_output/time.log
expire_in: 1 week expire_in: 1 week
except: except:
- /^wip\/.*/ - /^wip\/.*/
...@@ -34,45 +104,40 @@ integration.test: ...@@ -34,45 +104,40 @@ integration.test:
- integration - integration
``` ```
Standalone usage: example with the nrm source in `../`, provision the dependencies using: ### Argotk.hs usage:
```{.bash}
nix-shell -A test --arg nrm-src ../nrm
```
Then use the tool:
```{.bash} ```{.bash}
./argotk.hs --help argotk.hs --help
``` ```
Output: Output:
```{.txt pipe="sh"} ```{.txt pipe="sh"}
root/argotk.hs --help root/argotk/argotk.hs --help
``` ```
```{.bash} ```{.bash}
./argotk.hs helloworld --help argotk.hs helloworld --help
``` ```
```{.txt pipe="sh"} ```{.txt pipe="sh"}
root/argotk.hs helloworld --help root/argotk/argotk.hs helloworld --help
``` ```
```{.bash} ```{.bash}
./argotk.hs application --help argotk.hs application --help
``` ```
```{.txt pipe="sh"} ```{.txt pipe="sh"}
root/argotk.hs application --help root/argotk/argotk.hs application --help
``` ```
```{.bash} ```{.bash}
./argotk.hs daemon --help argotk.hs daemon --help
``` ```
```{.txt pipe="sh"} ```{.txt pipe="sh"}
root/argotk.hs daemon --help root/argotk/argotk.hs daemon --help
``` ```
##### Hacking ### Hacking
- edit `.README.md` in place of README.md. - edit `.README.md` in place of README.md.
- the ./shake.hs build file takes care of a few things for the development workflow (readme and completion generation). - the ./shake.hs build file takes care of a few things for the development
workflow (readme and completion generation).
...@@ -10,14 +10,87 @@ version(s), as part of development or continuous integration. This ...@@ -10,14 +10,87 @@ version(s), as part of development or continuous integration. This
gitlab CI snippets shows how to do this on a nix-enabled runner. Mind gitlab CI snippets shows how to do this on a nix-enabled runner. Mind
that this setup tracks argotest master. that this setup tracks argotest master.
### Usage
- Get Nix: `curl https://nixos.org/nix/install | sh`
- Use the 'argotest' nix function to provide an environment, among
which the `argotk.hs` tool, which runs tests. For example, we can
run default tests using pinned `argopkgs` sources like so:
``` {.nix}
nix-shell -E '{ argotest ? (builtins.fetchGit {
url = https://xgitlab.cels.anl.gov/argo/argotest.git;
ref="master";})
}: (import argotest {}).test' --run 'argotk.hs helloworld'
```
Alternatively, one can get rid of the `--run` option in order to be
provided with an environment in which all the argo components should be
in the `PATH`, along with the `argotk.hs` tool.
The arguments to the argotest function are defined in the default.nix
file in this repository. They all have default values.
For a more involved example, let's run a test in a custom environment.
Here, we'll run the 'helloworld' test in an environment that uses a
local `nrm` source, the master `libnrm` branch and a specific revision
of the `containers` branch:
``` {.nix}
nix-shell -E '{ argotest ? (builtins.fetchGit {
url = https://xgitlab.cels.anl.gov/argo/argotest.git;
ref="argo-nodeos-config-suid-dir";})
}:
(import argotest {
nrm-src = /path/to/nrm;
libnrm-src = builtins.fetchGit {
url = https://xgitlab.cels.anl.gov/argo/libnrm.git;
ref="master"; };
containers-src = builtins.fetchGit {
url = https://xgitlab.cels.anl.gov/argo/containers.git;
ref="fancy-branch-name";
rev="commit-revisions-string";
};
}).test' --run 'argotk.hs helloworld'
```
*WARNING* There are a few things one has to be aware of using this
workflow:
- Builds may fail if the local source repositories are dirty with old
build files.
- Builds may fail if `PWD` is in a `nosuid`-enabled filesystem.
- Without using the `rev` argument, the `builtins.fetchGit` nix
command prefetches and buffers its output, with an expiration time
that ranges ten minutes by default. Use a local checkout if you need
to modify some of these sources on the fly:
``` {.nix}
nix-shell -E '{ argotest ? /path/to/argotest }:
(import argotest {
nrm-src = /path/to/nrm;
# libnrm-src left at default argument value (inherits from argopkgs)
# containers-src left at default argument value (inherits from argopkgs)
}).test' --run 'argotk.hs helloworld'
```
### Example CI setup
For example, we could setup the CI for the `containers` repo like:
``` {.yml} ``` {.yml}
integration.test: integration.test:
stage: test stage: test
script: script:
- BUILD=$PWD - nix-shell -E '{ argotest ? (builtins.fetchGit {
- git clone https://xgitlab.cels.anl.gov/argo/argotest.git url = https://xgitlab.cels.anl.gov/argo/argotest.git;
- cd argotest ref="master";})
- nix-shell -A test --run "./argotk.hs helloworld" --arg nrm-src ../. }:
(import argotest { containers-src = ./. ; }).test'
--run 'argotk.hs helloworld'
artifacts: artifacts:
paths: paths:
- argotest/_output/cmd_err.log - argotest/_output/cmd_err.log
...@@ -34,17 +107,10 @@ integration.test: ...@@ -34,17 +107,10 @@ integration.test:
- integration - integration
``` ```
Standalone usage: example with the nrm source in `../`, provision the ### Argotk.hs usage:
dependencies using:
``` {.bash} ``` {.bash}
nix-shell -A test --arg nrm-src ../nrm argotk.hs --help
```
Then use the tool:
``` {.bash}
./argotk.hs --help
``` ```
Output: Output:
...@@ -67,7 +133,7 @@ Available commands: ...@@ -67,7 +133,7 @@ Available commands:
``` ```
``` {.bash} ``` {.bash}
./argotk.hs helloworld --help argotk.hs helloworld --help
``` ```
``` {.txt} ``` {.txt}
...@@ -79,19 +145,36 @@ Available options: ...@@ -79,19 +145,36 @@ Available options:
``` ```
``` {.bash} ``` {.bash}
./argotk.hs application --help argotk.hs application --help
``` ```
``` {.txt} ``` {.txt}
Usage: argotk.hs application COMMAND Usage: argotk.hs application [-w|--working_directory DIRECTORY] COMMAND
Setup stack and run an arbitrary command in a container. Setup stack and run an arbitrary command in a container.
Available options: Available options:
-w,--working_directory DIRECTORY
Working directory. Will be used for logging.
COMMAND Application to run inside the container COMMAND Application to run inside the container
-h,--help Show this help text -h,--help Show this help text
``` ```
##### Hacking ``` {.bash}
argotk.hs daemon --help
```
``` {.txt}
Usage: argotk.hs daemon [-w|--working_directory DIRECTORY]
Set up and launch the daemon in synchronous mode, with properly cleaned
sockets, logfiles.
Available options:
-w,--working_directory DIRECTORY
Working directory. Will be used for logging.
-h,--help Show this help text
```
### Hacking
- edit `.README.md` in place of README.md. - edit `.README.md` in place of README.md.
......
...@@ -16,7 +16,6 @@ import System.Console.ANSI.Types ( Color ) ...@@ -16,7 +16,6 @@ import System.Console.ANSI.Types ( Color )
import Options.Applicative import Options.Applicative
import System.Posix.Signals import System.Posix.Signals
import Control.Monad import Control.Monad
import System.Environment.FindBin
wdOption = strOption wdOption = strOption
( long "working_directory" ( long "working_directory"
......
...@@ -8,7 +8,7 @@ _argotk.hs() ...@@ -8,7 +8,7 @@ _argotk.hs()
CMDLINE=(${CMDLINE[@]} --bash-completion-word $arg) CMDLINE=(${CMDLINE[@]} --bash-completion-word $arg)
done done
COMPREPLY=( $(./argotk.hs "${CMDLINE[@]}") ) COMPREPLY=( $(./argotk/argotk.hs "${CMDLINE[@]}") )
} }
complete -o filenames -F _argotk.hs argotk.hs complete -o filenames -F _argotk.hs argotk.hs
...@@ -66,16 +66,15 @@ in rec ...@@ -66,16 +66,15 @@ in rec
cp ${manifests}/* $out/share/ cp ${manifests}/* $out/share/
''; '';
buildInputs = [ buildInputs = [
(hpkgs.ghcWithPackages (p: [p.argo p.FindBin])) (hpkgs.ghcWithPackages (p: [p.argo ]))
]; ];
propagatedBuildInputs = [ propagatedBuildInputs = [
(hpkgs.ghcWithPackages (p: [p.argo p.FindBin])) (hpkgs.ghcWithPackages (p: [p.argo ]))
pkgs.coreutils pkgs.coreutils
]; ];
inherit shellHook; inherit shellHook;
}; };
test = pkgs.stdenv.mkDerivation rec { test = pkgs.stdenv.mkDerivation rec {
name = "env"; name = "env";
env = pkgs.buildEnv { name = name; paths = buildInputs; }; env = pkgs.buildEnv { name = name; paths = buildInputs; };
...@@ -90,5 +89,23 @@ in rec ...@@ -90,5 +89,23 @@ in rec
MANIFESTS = ./manifests; MANIFESTS = ./manifests;
}; };
hack = pkgs.stdenv.mkDerivation rec {
name = "env";
env = pkgs.buildEnv { name = name; paths = buildInputs; };
buildInputs = [
containers
amg
nrm
(hpkgs.ghcWithPackages (p: devHPackages ++ [
argotk
argo
(pkgs.haskell.lib.doJailbreak p.panpipe)
p.turtle]))
];
inherit shellHook;
MANIFESTS = ./manifests;
};
} }
...@@ -12,7 +12,7 @@ main = shakeArgs shakeOptions $ do ...@@ -12,7 +12,7 @@ main = shakeArgs shakeOptions $ do
want ["README.md", "completion/argotk.sh"] want ["README.md", "completion/argotk.sh"]
"completion/*.sh" "completion/*.sh"
%> \out -> mkCompletionRule out "bash" $ takeFileName out -<.> "hs" %> \out -> mkCompletionRule out "bash" $ "argotk" </> takeFileName out -<.> "hs"
"README.md" %> \out -> do "README.md" %> \out -> do
let template = ".README.md" let template = ".README.md"
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment