feat(dev-new): cut lab over to the central OneCLI, retire the sidecar #352

Merged

dominik.polakovics merged 1 commit from afk/338 into main

2026-08-14 23:00:58 +02:00

dominik.polakovics commented

2026-08-14 22:14:21 +02:00

Owner

PR 2 of 2 for the OneCLI centralisation (follows #337 / ADR-0026, which named this "the follow-up PR's concern").

lab on dev-new now consumes the central OneCLI on web-arm instead of its own local sidecar, and the sidecar plus every trace of its dashboard path is deleted.

Repointed (`hosts/dev-new/modules/coding-lab/default.nix`)

onecli = {
  url = "https://onecli.cloonar.com";
  gatewayUrl = "http://10.42.98.10:10255";
  apiKeyFile = config.sops.secrets."coding-lab-onecli-api-key".path;
  caFile = "${./onecli-gateway-ca.pem}";
  dashboard = "off";
};

REST rides TLS, the gateway does not. url goes through web-arm's onecli.cloonar.com nginx vhost; the gateway is a CONNECT forward proxy that cannot be reverse-proxied, so runs dial 10.42.98.10:10255 (web-arm's wg_cloonar address) directly.
apiKeyFile is the coding-lab-onecli-api-key sops secret already present on dev-new (commit fc57bd7), declared with owner = config.services.lab.user; mode = "0400" — lab refuses any key file with group/other bits set. The local-mode auto-key-fetch oneshot is gone; it only ever worked because local mode minted keys unauthenticated.
caFile is the gateway's interception CA, fetched once from https://onecli.cloonar.com/v1/gateway/ca and committed as a plain PEM (CN=OneCLI Local Gateway CA, valid to 2036-08-11). Referenced as a store path, so it is deterministic with no boot-time fetch. A note beside it records that a OneCLI reinstall mints a new CA and this file must then be refetched.
dashboard = "off" and dashboardAddr dropped entirely — the central dashboard has its own vhost, and lab refuses to start if dashboardAddr is set outside port mode.

Demolished

hosts/dev-new/modules/coding-lab/onecli.nix deleted: both podman containers, the init-onecli-network unit, the coding-lab-onecli-key fetch oneshot, and dev-new's :8081 firewall opening. It was dev-new's only consumer of virtualisation.oci-containers.
onecli-dashboard vhost removed from hosts/fw/vms/web/lab.nix.
8081 hairpin DNAT removed from hosts/fw/modules/firewall.nix.
8081 dropped from the web microvm's allowedTCPPorts, restoring the [ 22 80 443 ] that commit 073f15f had extended for exactly this vhost.
One-line comment fix in hosts/fw/vms/web/matrix.nix, which explained masPort = 8083 by pointing at the now-deleted vhost.

Incidentally this also clears a latent bind conflict: the deleted vhost bound 0.0.0.0:8081 while piped publishes 127.0.0.1:8081 on the same VM.

Accepted consequences

A brief fail-closed spawn window while bento switches dev-new: the old sidecar stops before the new config is live, and lab fails a spawn closed rather than running an agent without the gateway.
From then on, no agent run spawns while the fw↔web-arm wg_cloonar tunnel is down. The tunnel — and web-arm's availability — is now production-critical for lab. This is ADR-0026's stated consequence, fail-closed by design; the alternative is a public credential gateway.
Grants and secrets do not migrate. The sidecar's entries must be re-entered by hand in the central dashboard (it is a week old, so a handful). lab re-creates its per-repo agent identities lazily on first spawn.

Operator follow-ups

Orphaned podman volumes on dev-new: onecli-pgdata and onecli-appdata are left behind and can be pruned — dev-new is cattle, nothing to preserve.
Stale secret on web-arm: an unused copy of coding-lab-onecli-api-key sits in hosts/web-arm/secrets.sops.yaml and can be removed. No sops file was touched by this PR.

Verification

scripts/test-configuration dev-new and scripts/test-configuration fw both evaluate green (run by the pre-commit hook on this commit).
Option semantics checked against the pinned coding-lab rev 47f57bbd4011: dashboard is enum [ "off" "port" "subdomain" ]; apiKeyFile is refused unless 0600-or-stricter; caFile is read at spawn and concatenated after the system roots into a per-run 0644 bundle exported as SSL_CERT_FILE / NODE_EXTRA_CA_CERTS / REQUESTS_CA_BUNDLE / GIT_SSL_CAINFO; gatewayUrl becomes HTTPS_PROXY in each spawned run.
Network path traced in-repo: fw's dnsmasq already resolves onecli.cloonar.com to 10.42.98.10, fw forwards server→wg_cloonar, and web-arm admits 443 globally plus 10255/10256 on the tunnel interface. Note that fw masquerades tunnel egress, so web-arm sees these requests from 10.42.98.1, which is inside its allow-list.
Post-deploy checks for the operator: a spawned run gets HTTPS_PROXY=http://…@10.42.98.10:10255 and a granted service call succeeds; GET /api/v1/onecli/health on lab reports ok; the dashboard answers at https://onecli.cloonar.com and nothing answers on 8081.

Closes #338

PR 2 of 2 for the OneCLI centralisation (follows #337 / ADR-0026, which named this "the follow-up PR's concern"). lab on `dev-new` now consumes the central OneCLI on `web-arm` instead of its own local sidecar, and the sidecar plus every trace of its dashboard path is deleted. ## Repointed (`hosts/dev-new/modules/coding-lab/default.nix`) ```nix onecli = { url = "https://onecli.cloonar.com"; gatewayUrl = "http://10.42.98.10:10255"; apiKeyFile = config.sops.secrets."coding-lab-onecli-api-key".path; caFile = "${./onecli-gateway-ca.pem}"; dashboard = "off"; }; ``` - **REST rides TLS, the gateway does not.** `url` goes through web-arm's `onecli.cloonar.com` nginx vhost; the gateway is a CONNECT forward proxy that cannot be reverse-proxied, so runs dial `10.42.98.10:10255` (web-arm's `wg_cloonar` address) directly. - **`apiKeyFile`** is the `coding-lab-onecli-api-key` sops secret already present on dev-new (commit fc57bd7), declared with `owner = config.services.lab.user; mode = "0400"` — lab refuses any key file with group/other bits set. The local-mode auto-key-fetch oneshot is gone; it only ever worked because local mode minted keys unauthenticated. - **`caFile`** is the gateway's interception CA, fetched once from `https://onecli.cloonar.com/v1/gateway/ca` and committed as a plain PEM (`CN=OneCLI Local Gateway CA`, valid to 2036-08-11). Referenced as a store path, so it is deterministic with no boot-time fetch. A note beside it records that a OneCLI reinstall mints a new CA and this file must then be refetched. - **`dashboard = "off"`** and `dashboardAddr` dropped entirely — the central dashboard has its own vhost, and lab refuses to start if `dashboardAddr` is set outside port mode. ## Demolished - `hosts/dev-new/modules/coding-lab/onecli.nix` deleted: both podman containers, the `init-onecli-network` unit, the `coding-lab-onecli-key` fetch oneshot, and dev-new's `:8081` firewall opening. It was dev-new's only consumer of `virtualisation.oci-containers`. - `onecli-dashboard` vhost removed from `hosts/fw/vms/web/lab.nix`. - 8081 hairpin DNAT removed from `hosts/fw/modules/firewall.nix`. - 8081 dropped from the web microvm's `allowedTCPPorts`, restoring the `[ 22 80 443 ]` that commit 073f15f had extended for exactly this vhost. - One-line comment fix in `hosts/fw/vms/web/matrix.nix`, which explained `masPort = 8083` by pointing at the now-deleted vhost. Incidentally this also clears a latent bind conflict: the deleted vhost bound `0.0.0.0:8081` while piped publishes `127.0.0.1:8081` on the same VM. ## Accepted consequences - **A brief fail-closed spawn window** while bento switches dev-new: the old sidecar stops before the new config is live, and lab fails a spawn closed rather than running an agent without the gateway. - **From then on, no agent run spawns while the fw↔web-arm `wg_cloonar` tunnel is down.** The tunnel — and web-arm's availability — is now production-critical for lab. This is ADR-0026's stated consequence, fail-closed by design; the alternative is a public credential gateway. - **Grants and secrets do not migrate.** The sidecar's entries must be re-entered by hand in the central dashboard (it is a week old, so a handful). lab re-creates its per-repo agent identities lazily on first spawn. ## Operator follow-ups - **Orphaned podman volumes on dev-new**: `onecli-pgdata` and `onecli-appdata` are left behind and can be pruned — dev-new is cattle, nothing to preserve. - **Stale secret on web-arm**: an unused copy of `coding-lab-onecli-api-key` sits in `hosts/web-arm/secrets.sops.yaml` and can be removed. No sops file was touched by this PR. ## Verification - `scripts/test-configuration dev-new` and `scripts/test-configuration fw` both evaluate green (run by the pre-commit hook on this commit). - Option semantics checked against the pinned coding-lab rev `47f57bbd4011`: `dashboard` is `enum [ "off" "port" "subdomain" ]`; `apiKeyFile` is refused unless `0600`-or-stricter; `caFile` is read at spawn and concatenated after the system roots into a per-run `0644` bundle exported as `SSL_CERT_FILE` / `NODE_EXTRA_CA_CERTS` / `REQUESTS_CA_BUNDLE` / `GIT_SSL_CAINFO`; `gatewayUrl` becomes `HTTPS_PROXY` in each spawned run. - Network path traced in-repo: fw's dnsmasq already resolves `onecli.cloonar.com` to `10.42.98.10`, fw forwards `server`→`wg_cloonar`, and web-arm admits 443 globally plus 10255/10256 on the tunnel interface. Note that fw masquerades tunnel egress, so web-arm sees these requests from `10.42.98.1`, which is inside its allow-list. - Post-deploy checks for the operator: a spawned run gets `HTTPS_PROXY=http://…@10.42.98.10:10255` and a granted service call succeeds; `GET /api/v1/onecli/health` on lab reports ok; the dashboard answers at `https://onecli.cloonar.com` and nothing answers on 8081. Closes #338

dominik.polakovics added 1 commit

2026-08-14 22:14:21 +02:00

feat(dev-new): cut lab over to the central OneCLI, retire the sidecar f8a0617e4c

lab on dev-new now points at the central instance on web-arm (ADR-0026)
instead of its own local sidecar: REST at https://onecli.cloonar.com over
that host's nginx TLS vhost, and the gateway dialled directly at
10.42.98.10:10255 over wg_cloonar, since a CONNECT forward proxy cannot be
reverse-proxied. The API key comes from a sops secret rather than local
mode's unauthenticated auto-minting, and the gateway's interception CA is
committed as a plain PEM so there is no boot-time fetch.

With that, the sidecar and its whole dashboard path go: both podman
containers, the podman network unit, the key/CA fetch oneshot, dev-new's
:8081 opening, web-02's onecli-dashboard vhost and fw's 8081 hairpin DNAT.
The central dashboard has its own vhost, so lab's port-mode proxy is
obsolete and dashboard is off.

Closes #338

dominik.polakovics commented

2026-08-14 22:32:05 +02:00

Author

Owner

[autoland] verdict: pass

This was generated by AI while landing a PR.

PASS — validated locally; no CI signal existed to rely on.

Signal relied on: labctl pr checks reported state: none, so nothing vouched and the repo's own gate was run against the PR head (f8a0617):

scripts/test-configuration dev-new → completed successfully (exit 0)
scripts/test-configuration fw → completed successfully (exit 0)

Both hosts evaluate clean, warnings limited to the usual --add-root notice.

Conventions: Conventional Commits title, Closes #338 present and resolving. Head sits directly on current main (fc57bd7) — no conflict.

Checks beyond the build:

The committed onecli-gateway-ca.pem is byte-identical to what https://onecli.cloonar.com/v1/gateway/ca serves right now, so the pinned trust material matches the live gateway.
Its missing trailing newline is harmless: writeTrustBundle (coding-lab internal/instance/gateway.go:421-424) writes the system roots trimmed, then a newline, then the gateway PEM last — nothing is concatenated after it.
Option semantics match the pinned rev 47f57bbd4011: dashboard is enum [ "off" "port" "subdomain" ], caFile/gatewayUrl are nullOr str, and --onecli-dashboard is only emitted when non-off. apiKeyFile is wired owner = services.lab.user; mode = "0400", satisfying the 0600-or-stricter contract.
coding-lab-onecli-api-key exists in hosts/dev-new/secrets.sops.yaml; no sops file was modified.
Deletions leave nothing dangling: no oci-containers/podman references remain under hosts/dev-new/, and no 8081 reference remains on the web VM (piped.nix binds 127.0.0.1:8081, which the removed 0.0.0.0 vhost had been shadowing).
Network path confirmed in-repo: fw's dnsmasq maps onecli.cloonar.com → 10.42.98.10, and web-arm publishes the gateway on 10.42.98.10:10255 with networking.firewall.interfaces."wg_cloonar".allowedTCPPorts = [ 10255 10256 ].

Diff scope matches issue #338 exactly; the two additions beyond its file list (the web VM's allowedTCPPorts and the matrix.nix comment) are direct consequences of the removals, not drive-by changes.

No CONCERNS. The fail-closed tunnel dependency and the non-migrating grants are ADR-0026's stated, accepted consequences, already recorded in the PR body.

[autoland] verdict: pass > *This was generated by AI while landing a PR.* **PASS** — validated locally; no CI signal existed to rely on. **Signal relied on:** `labctl pr checks` reported `state: none`, so nothing vouched and the repo's own gate was run against the PR head (f8a0617): - `scripts/test-configuration dev-new` → `completed successfully` (exit 0) - `scripts/test-configuration fw` → `completed successfully` (exit 0) Both hosts evaluate clean, warnings limited to the usual `--add-root` notice. **Conventions:** Conventional Commits title, `Closes #338` present and resolving. Head sits directly on current `main` (fc57bd7) — no conflict. **Checks beyond the build:** - The committed `onecli-gateway-ca.pem` is **byte-identical** to what `https://onecli.cloonar.com/v1/gateway/ca` serves right now, so the pinned trust material matches the live gateway. - Its missing trailing newline is harmless: `writeTrustBundle` (coding-lab `internal/instance/gateway.go:421-424`) writes the system roots trimmed, then a newline, then the gateway PEM last — nothing is concatenated after it. - Option semantics match the pinned rev 47f57bbd4011: `dashboard` is `enum [ "off" "port" "subdomain" ]`, `caFile`/`gatewayUrl` are `nullOr str`, and `--onecli-dashboard` is only emitted when non-`off`. `apiKeyFile` is wired `owner = services.lab.user; mode = "0400"`, satisfying the 0600-or-stricter contract. - `coding-lab-onecli-api-key` exists in `hosts/dev-new/secrets.sops.yaml`; no sops file was modified. - Deletions leave nothing dangling: no `oci-containers`/podman references remain under `hosts/dev-new/`, and no 8081 reference remains on the web VM (`piped.nix` binds 127.0.0.1:8081, which the removed 0.0.0.0 vhost had been shadowing). - Network path confirmed in-repo: fw's dnsmasq maps `onecli.cloonar.com` → `10.42.98.10`, and web-arm publishes the gateway on `10.42.98.10:10255` with `networking.firewall.interfaces."wg_cloonar".allowedTCPPorts = [ 10255 10256 ]`. Diff scope matches issue #338 exactly; the two additions beyond its file list (the web VM's `allowedTCPPorts` and the `matrix.nix` comment) are direct consequences of the removals, not drive-by changes. No CONCERNS. The fail-closed tunnel dependency and the non-migrating grants are ADR-0026's stated, accepted consequences, already recorded in the PR body.

dominik.polakovics merged commit f552d41af4 into main

2026-08-14 23:00:58 +02:00

dominik.polakovics deleted branch afk/338

2026-08-14 23:00:58 +02:00

dominik.polakovics referenced this pull request from a commit

2026-08-14 23:00:59 +02:00

Merge pull request 'feat(dev-new): cut lab over to the central OneCLI, retire the sidecar' (#352) from afk/338 into main