ci: warm the nix store (persistent runner / binary cache) to cut the hermetic gate to ~2-4 min #33

New issue

Open

opened 2026-07-07 23:24:54 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-07-07 23:24:54 +02:00

Owner

Needs triage — decision + owner TBD. Sibling of #32 (split CI), which reduces how often this gate runs; this issue makes the gate itself fast. Cross-repo: options 1–2 touch Cloonar/nixos (the runner host), not just this repo.

Problem

The hermetic nix flake check gate costs ~12–13 min/run because the Docker-backed forgejo-runner is ephemeral — /nix is wiped each job, so the full closure (Go toolchain, node, golangci-lint, ~160 MB npm deps, Go vendor modules) is re-downloaded + rebuilt cold every run. The nix install itself is only ~2 min (bump-nixos-pin is the natural control); the remaining ~10 min is pure cold-store cache misses. There is zero store reuse across runs.

Even after #32 (split CI) skips this gate on most PRs, it still runs on every nix/dep change, on bump-nixos-pin → test-configuration, and across the whole Cloonar/nixos fleet — so a warm store keeps standing value independent of #32.

Options to weigh (not decided)

Persistent store on the self-hosted runner — run the job natively on the NixOS runner host (runner label → host, not docker) using its own nix + permanent /nix, or bind-mount a persistent host /nix (+ Go/npm caches) into the job container via forgejo-runner config.yml. Biggest win, no new service, most nix-native; needs a runner-config change in Cloonar/nixos. Steady state ~2–4 min (incremental rebuilds only). Concurrency: a shared /nix wants the nix daemon to arbitrate (the host already runs one) rather than the daemonless --init none install this repo's workflow uses.
Self-hosted binary cache (Attic / nix-serve-ng) — runners pull prebuilt store paths instead of rebuilding; also speeds the deploy and the whole Cloonar/nixos fleet. More infra: a cache service + signing key (sops-managed) + extra-substituters / trusted-public-keys wiring on every consumer.
Prebaked OCI image (dockerTools.buildLayeredImage carrying the build closure, hosted on Forgejo's built-in container registry, rebuilt on pin bump) — stateless, but needs rebuild discipline and is strictly worse than (1) at steady state. This is the "make a docker image" idea; note it only helps by carrying the store, not just nix.

Notes for triage

Don't parallelize flake check into multiple jobs until the store is warm/shared — on cold ephemeral runners N jobs each re-pay the full cold tax (net-worse). Parallelism only pays after (1) or (2).
Decide: which lever (or 1 + 2 together), and whether it complements or supersedes #32 — if the split already makes the nix gate rare, a full cache server may be overkill vs. just a persistent /nix.
Likely owner = human/maintainer (touches Cloonar/nixos infra) → candidate ready-for-human after triage, not an AFK agent.
Same lever also cuts the bump-nixos-pin deploy path and any future store-postgres job.

> **Needs triage** — decision + owner TBD. Sibling of #32 (split CI), which reduces *how often* this gate runs; this issue makes the gate *itself* fast. Cross-repo: options 1–2 touch `Cloonar/nixos` (the runner host), not just this repo. ## Problem The hermetic `nix flake check` gate costs **~12–13 min/run** because the Docker-backed forgejo-runner is **ephemeral** — `/nix` is wiped each job, so the full closure (Go toolchain, node, golangci-lint, ~160 MB npm deps, Go vendor modules) is re-downloaded + rebuilt cold every run. The nix install itself is only ~2 min (`bump-nixos-pin` is the natural control); the remaining ~10 min is pure cold-store cache misses. There is **zero store reuse** across runs. Even after #32 (split CI) skips this gate on most PRs, it still runs on every nix/dep change, on `bump-nixos-pin` → `test-configuration`, and across the whole `Cloonar/nixos` fleet — so a warm store keeps standing value independent of #32. ## Options to weigh (not decided) 1. **Persistent store on the self-hosted runner** — run the job natively on the NixOS runner host (runner label → host, not docker) using its own nix + permanent `/nix`, or bind-mount a persistent host `/nix` (+ Go/npm caches) into the job container via forgejo-runner `config.yml`. **Biggest win, no new service**, most nix-native; needs a runner-config change in `Cloonar/nixos`. Steady state ~2–4 min (incremental rebuilds only). Concurrency: a shared `/nix` wants the nix **daemon** to arbitrate (the host already runs one) rather than the daemonless `--init none` install this repo's workflow uses. 2. **Self-hosted binary cache (Attic / nix-serve-ng)** — runners *pull prebuilt store paths* instead of rebuilding; also speeds the deploy and the whole `Cloonar/nixos` fleet. More infra: a cache service + signing key (sops-managed) + `extra-substituters` / `trusted-public-keys` wiring on every consumer. 3. **Prebaked OCI image** (`dockerTools.buildLayeredImage` carrying the build closure, hosted on Forgejo's built-in container registry, rebuilt on pin bump) — stateless, but needs rebuild discipline and is strictly worse than (1) at steady state. This is the "make a docker image" idea; note it only helps by carrying the *store*, not just nix. ## Notes for triage - **Don't** parallelize `flake check` into multiple jobs until the store is warm/shared — on cold ephemeral runners N jobs each re-pay the full cold tax (net-worse). Parallelism only pays after (1) or (2). - Decide: which lever (or 1 + 2 together), and whether it **complements or supersedes** #32 — if the split already makes the nix gate rare, a full cache server may be overkill vs. just a persistent `/nix`. - Likely **owner = human/maintainer** (touches `Cloonar/nixos` infra) → candidate `ready-for-human` after triage, not an AFK agent. - Same lever also cuts the `bump-nixos-pin` deploy path and any future `store-postgres` job.

dominik.polakovics added the

needs-triage

enhancement

labels

2026-07-07 23:24:54 +02:00

dominik.polakovics referenced this issue

2026-07-07 23:25:27 +02:00

ci: split CI — fast native gate every PR, hermetic nix flake check only on nix/dep changes #32

dominik.polakovics referenced this issue

2026-07-10 01:30:47 +02:00

Provider conformance suite + provider-author checklist: executable seam obligations, live-spike merge gate #80