Continuous integration

This guide is for maintainers changing GitHub Actions, test tiers, self-hosted runners, or branch protection. For ordinary local test commands, start with the Quick start.

Why there are two kinds of job

Every shipped preset compiles with NVHPC's mpic++ and mpifort against TPLs installed on the host (/raid/mog/libs_VVM_cpu), and the reference data every test compares against was generated by that toolchain. A GitHub-hosted runner has none of it. Building VVMex there is not impossible — see Could CI run entirely on GitHub? — but it would gate a different compiler than the one the project ships, which is not the same check.

So CI is split by what a job needs:

Workflow	Runner	Trigger	Cost
`checks.yml`	GitHub-hosted	every push and PR, forks included	~1 min
`cpu-tests.yml`	self-hosted	every push and PR from this repo	~11 min
`gpu-tests.yml`	self-hosted	every push and PR from this repo	~14 min
`nightly.yml`	self-hosted	19:00 UTC daily, or manual	~10 min GPU, ~1.5 h CPU
`docs.yml`	GitHub-hosted	push to `main` touching `docs/`	~1 min

checks.yml is the only one that runs without a self-hosted runner, and the only one a fork's pull request can trigger.

What each job checks

`checks.yml` — hermetic

Python — ruff check --select E9,F63,F7,F82 over submit.py, tools/*.py and tests/scripts/*.py: syntax errors and undefined names, not style. The full default rule set reports 42 nits on this tree, which is not worth blocking a merge on.
Configs — all 114 shipped JSON and YAML configurations parse, and every extends names a file that exists. Worth gating because the model silently ignores unknown keys, so a parse failure is one of the few configuration errors it will not absorb.
Test registry — tests/scripts/check_test_registry.py reads tests/CMakeLists.txt and checks that every file it names exists: unit-test sources, tests/configs/*.json, per-backend baselines, and references/ digests. It also reports the reverse — data no registered test reads. CMake resolves these at configure time and CTest only at run time, so a case registered without its config, or a digest left behind by a removed test, otherwise surfaces long after the commit that caused it.
Shell — bash -n plus shellcheck --severity=error.
Docs — mkdocs build --strict, which fails on a broken internal link or a nav entry pointing at a missing page. The deploy workflow does not use --strict, so without this a bad link reaches the published site.
Hygiene — files added by a pull request that exceed 1 MB must be LFS-tracked. Only added files are checked: tests/baselines_cpu/*.h5 (7 MB) predates the LFS rules and is not tracked.

Run the same checks locally:

python3 tests/scripts/check_test_registry.py
mkdocs build --strict
ruff check --select E9,F63,F7,F82 submit.py tools/*.py tests/scripts/*.py

`cpu-tests.yml` — the pull-request gate

A clean CPU build plus ctest with no optional tier enabled, which is exactly the default tier: the unit tests, the HDF5 precision cases, and the seven dynamical-core regression cases against tests/baselines_cpu/.

Enabling a tier is a configure-time decision (see the tier block at the top of tests/CMakeLists.txt), so ctest with no arguments is already the right selection — there is no need to filter by label.

The tree is deleted before configuring, every run. Two defects make a reused tree unsafe: CMakeLists.txt FORCE-writes empty OpenMP_*_FLAGS into the cache so a second configure cannot find OpenMP (after which cmake --build returns 0 having built nothing), and there is no header dependency tracking, so a reused tree can keep objects holding a stale class layout.

`gpu-tests.yml` — the reference backend

The same shape as the CPU gate, on a runner labelled vvmex-gpu, running the default and physics tiers. The physics tier is the reason it exists: P3, RRTMGP and the Noah land model are checked nowhere else, because every digest under tests/references/ was generated on a GPU and tests/references_cpu/ has one case.

It refuses outright if it lands on a runner whose VVM_CI_BACKEND is not gpu, rather than building the wrong tree.

It checks out with lfs: false and then runs git lfs pull. actions/checkout implements lfs: true by running git lfs install --local, which refuses when a pre-push hook it did not write already exists — and .githooks/pre-push is tracked, so it is in every clone. Overwriting it with --force would dirty the working tree, since it is a tracked file. Fetching the objects needs no hooks.

`nightly.yml` — the slow tiers

GPU — the whole suite with VVM_TEST_BP5, VVM_TEST_PHYSICS and VVM_TEST_MULTIRANK on. This is the reference backend: every digest under tests/references/ was generated here, and it is the only place the physics tier can run, since tests/references_cpu/ has digests for one case. It needs 4 physical GPUs for the multirank tier and takes about six minutes.
CPU full — the CPU tiers the PR gate leaves out, BP5 and multirank.

Measured tier costs on blaze, which is why the split falls where it does. CPU figures are at 64 threads per rank:

Tier	CPU	GPU
unit + precision	1.0 min	0.9 min
default cases	4.9 min	0.9 min
bp5	12.1 min (at 16 threads)	2.3 min
multirank	> 60 min	0.8 min

The PR gate — unit, precision and the default cases — is 27 tests in 3.5 min, plus about eight minutes to build from a deleted tree.

Those CPU figures are with tests running concurrently. On a CPU build the scarce resource is cores, not devices, so each test declares a PROCESSORS weight (ranks x threads) and ctest -j <cores> packs them onto the machine; the GPU build keeps a hard RESOURCE_LOCK because several NCCL ranks per device is unsupported. Measured on a 224-core host, default tier:

	wall time
serial, 16 threads/rank (the old default)	14 min
serial, 64 threads/rank	5.8 min
concurrent, 64 threads/rank	3.5 min

More threads per rank is not automatically better. At 56 threads the same suite takes 5.2 min, because CTest then fits a different number of tests side by side. The default cases are 32x32x33 and 512x16x74 grids — small enough that beyond a point a test only takes cores from the test that would have run beside it.

The CPU multirank tier is the outlier, and it is not a defect in the tests: the wind solver exchanges halos many times per step, and at 4 ranks over MPI on a 32² grid that is pure latency with no work to hide it. twisting takes about a second at one rank and 20 minutes at four. Keep it out of any per-commit gate.

Setting up a runner

Development happens on more than one machine, so nothing in the workflows names one. Everything machine-specific comes from a single CMake preset, and one script turns a preset into a registered runner.

Until a runner exists and VVMEX_RUNNER is set, cpu-tests.yml and nightly.yml do nothing at all — they are guarded on that variable, so merging them cannot leave pull requests waiting on a runner that is not there.

Per machine

export VVM_ROOT=/path/to/VVMex
tools/setup_ci_runner.sh --preset blaze-cpu --token <registration-token> \
                         --dir /raid/mog/actions-runner-blaze-cpu

Put --dir on a filesystem with room. The work directory holds a full checkout plus a build tree per run — about 300 MB for a CPU build and 9 GB for a GPU one — and the script warns if the target looks tight.

Without root, add --user-service:

tools/setup_ci_runner.sh --preset blaze-cpu --token <registration-token> \
                         --dir /raid/mog/actions-runner-blaze-cpu \
                         --user-service

This installs a systemd --user unit under ~/.config/systemd/user/ instead of calling svc.sh install, which needs sudo. It also runs loginctl enable-linger, which is what keeps a user unit alive after the last session ends and starts it again at boot — many systems let a user enable that for themselves. Manage it with systemctl --user {status,stop} vvmex-runner-<preset> and read its log with journalctl --user -u vvmex-runner-<preset> -f.

If linger cannot be enabled, the runner stops when you log out. --no-service plus ./run.sh in a tmux session is the last resort.

The token is on Settings → Actions → Runners → New self-hosted runner and expires after an hour. That is the whole per-machine procedure; the script:

refuses a preset that is not in CMakePresets.json, listing the ones that are;
reads the preset's VVM_ENABLE_GPU and labels the runner vvmex-cpu or vvmex-gpu accordingly, so a CPU box never picks up the GPU job;
downloads and configures actions/runner, naming it <host>-<preset>;
writes the runner's .env (below);
installs and starts the systemd service.

Add a second machine by running the same command there with its own --preset — nano5, twnia3, f1-cpu, whatever that host has. No workflow changes.

What lands in `.env`, and why it is generated

A runner started as a service reads no shell profile, so it has none of the environment a build needs. tools/ci_runner_env.py derives it from the preset:

Variable	From
`VVM_CI_PRESET`, `VVM_CI_BUILD_DIR`, `VVM_CI_BACKEND`	the preset's name, `binaryDir`, `VVM_ENABLE_GPU`
`VVM_CI_BUILD_JOBS`, `VVM_CI_CTEST_JOBS`	this machine's core count
`VVM_CI_TEST_THREADS`	half the cores, capped at 64
`PATH`	the preset's MPI `bin`, plus where `cmake`/`ctest`/`python3`/`git` actually are on this host
`LD_LIBRARY_PATH`	the preset's TPL prefixes first, then the NVHPC/HPC-X/CUDA runtime and the `--gcc-toolchain` prefix

Two details that are easy to get wrong by hand:

The caller's LD_LIBRARY_PATH is deliberately not inherited. submit.py layers preset directories on top of it, which is right interactively and wrong here: the shell doing the setup may have the other backend's prefix loaded, and CPU and CUDA Kokkos share a SONAME, so the runner would silently resolve the wrong library.
The CUDA runtime directory is included even for a CPU preset. HPC-X's OpenMPI is CUDA-aware, so a CPU vvm still needs libcudart.so.12. This was found by running the CPU binary in an environment containing nothing but the generated file — worth repeating on a new machine:

env -i $(grep -v '^#' /path/to/runner/.env | tr '\n' ' ') HOME=$HOME VVM_ROOT=$PWD \
  bash -c 'ldd $VVM_ROOT/build_cpu/vvm | grep "not found"'

One more entry is written unconditionally: DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1. The runner is a vendored .NET application that probes for ICU at startup, and LD_LIBRARY_PATH points at the scientific stack, which can carry a different ICU version than the system one — .NET then refuses to start with "Couldn't find a valid ICU package" and the service never picks up a job. Invariant globalization sidesteps the probe, and the runner needs no locale support.

Re-run tools/setup_ci_runner.sh --preset <name> --env-only after moving the TPLs or editing CMakePresets.json, then restart the service.

Once per repository

Set the variable for each backend you have registered a runner for (Settings → Secrets and variables → Actions → Variables):

Variable	Means	Gates
`VVMEX_RUNNER`	a CPU runner is registered	`cpu-tests.yml`, nightly's CPU job
`VVMEX_GPU_RUNNER`	a GPU runner is registered	`gpu-tests.yml`, nightly's GPU job

Each backend has its own variable because a job dispatched to a label no runner carries does not skip — it queues for 24 hours and then fails. Gating a GPU job on the CPU variable therefore turns "no GPU runner" into a failure every night rather than a quiet skip. Set only the variables whose runners exist, and unset one to take that backend's workflows out of service without deleting them.

Then add the required status checks under Settings → Branches for main — see What to require.

The README carries a badge for checks.yml only. Add one for the CPU gate once a runner is live — a badge for a workflow that has never run renders as "no status", which reads as broken.

Branch protection

Until this is configured, CI is advisory: a pull request shows red checks and the Merge button still works, and anyone can push straight to main. Branch protection is what turns the checks into a gate.

Settings → Branches → Add branch protection rule for main:

Require a pull request before merging
Require status checks to pass before merging

What to require

A check only appears in the search box after it has run at least once, so open one pull request first, let it finish, then come back and add them.

Check	Require	Why
`Python`	yes	seconds, no dependencies
`Configs and test registry`	yes	seconds, catches uncommitted test inputs
`Shell`	yes	seconds
`Docs build`	yes	seconds, catches broken links before they publish
`Build and test (CPU)`	yes, once the runner has been green a few times	the real correctness gate
`Build and test (GPU)`	not yet	see below
`Repository hygiene`	optional	only runs on pull requests, which is where it matters

Do not require the GPU check while it is still settling. Requiring it ties every merge to one machine having a working GPU runner; with the GPU tier only days old, an advisory red is more useful than a blocked repository. Promote it once it has run green across a week without intervention.

Two ways to deadlock a pull request

A required check that never reports. A workflow excluded by a path or branch filter produces no check run at all, so the requirement sits pending — not failed — and the pull request cannot proceed. This is why every workflow here filters paths-ignore on push only and leaves pull_request unfiltered. Do not add a path filter to a pull_request trigger for anything you require.

A required check whose runner is gone. Build and test (CPU) cannot pass without a machine carrying vvmex-cpu. If the runner is offline, merges stop. That is the intended trade for a self-hosted gate, but it is worth knowing before a deadline: taking the runner down blocks merging until it returns, or until the requirement is removed.

Bypass

"Do not allow bypassing the above settings" decides whether administrators are held to the rule. As the sole maintainer you may want the escape hatch; leaving it unchecked means the protection is a speed bump for you and a wall for everyone else. Either is defensible — just make it a decision rather than a default.

What this does not cover

Branch protection gates merges, server-side. .githooks/pre-push gates your pushes, from whatever state your build tree is in, and --no-verify skips it. Now that the CPU gate runs on every push, the hook's ctest is largely redundant with it; what the hook still does that nothing else does is run git lfs pre-push, which is how LFS objects get uploaded at all.

The fork problem

A self-hosted runner executes whatever a workflow says, on real hardware, with the runner's credentials. If a pull request from a fork could trigger it, anyone could run code on blaze by opening a pull request.

cpu-tests.yml and nightly.yml therefore refuse any pull request whose head repository is not this repository. Fork pull requests still get the whole checks.yml workflow, which runs on GitHub's infrastructure and touches nothing.

Do not "fix" a fork PR showing no CPU results by switching the trigger to pull_request_target — that runs the base branch's workflow with write access to the repository, which is the exact hole this guard closes. Review the diff and re-run the job from a branch in this repository instead.

Could CI run entirely on GitHub?

Partly. The build is not the hard part — the reference data is.

What would work. CMakeLists.txt already carries a GNU Fortran branch (-ffree-line-length-none -fdefault-real-8 …), so gfortran is contemplated by the build system rather than rejected. Kokkos-OpenMP, ADIOS2 and PnetCDF are not all in apt, but they build from source once into a container image on GHCR, and after that a hosted job just pulls it. A GCC CPU build of VVMex on GitHub is a realistic engineering task.

What would not. Every CPU test compares against data generated by the NVHPC CPU build:

the six tests/baselines_cpu/*.h5 cases use a 1e-6 tolerance, so a GCC build might pass them;
tests/references_cpu/mountain.json is a SHA-256 digest, and a different compiler changes FMA contraction and vectorization, so it would not.

Making it green means generating and maintaining a third reference set (baselines_gcc/, references_gcc/) that gates a compiler you do not ship with. Green CI on GCC would not be evidence that the NVHPC build is correct, which is the property the bit-for-bit rules exist to protect.

-DVVM_DETERMINISTIC_FP=ON may narrow this for the dry dynamical core — it already makes CPU and GPU agree bit-for-bit there — but that is untested across compilers, and physics runs can never match.

The other two limits.

Time. A standard hosted runner is 4 vCPU (2 for private repos). VVMex plus the EAMxx physics stack builds in under ten minutes at -j64 on blaze; at 4 cores expect well over an hour. ccache in the Actions cache helps, but this repo has no header dependency tracking, so a touched header invalidates effectively everything.
No GPU. GitHub-hosted runners have none, and the reference backend is the GPU: every digest in tests/references/ and the whole physics tier are GPU-only. That half can never move to GitHub at any price.

If maintaining a runner is the objection, not the money, GitHub sells hosted larger runners up to 64 cores. A 64-core Linux runner is billed per minute, so a ~15-minute build-and-test job costs on the order of a dollar. That solves the time problem and nothing else: the compiler and reference-data problems and the missing GPU are unchanged.

Recommendation. Keep the split as it is. Hermetic checks on GitHub, build and test on blaze. This is what HPC codes with compiler-specific reference data generally do, and it is the arrangement in which a green tick means the thing you actually ship is correct.

The pre-push hook is not a substitute

.githooks/pre-push runs ctest in build/ before every push. It gates your pushes, on whatever state your build tree happens to be in, and it is skipped entirely with --no-verify. CI runs from a clean tree on a known configuration and gates merges. Keep both; they are not the same check.