Cloonar/nixos

Fork 0

dev-new/fw: cut lab over to the central OneCLI on web-arm and retire the sidecar #338

New issue

Closed

opened 2026-08-14 12:38:50 +02:00 by dominik.polakovics · 1 comment

dominik.polakovics commented

2026-08-14 12:38:50 +02:00

Owner

Keep as needs-triage until those prerequisites are done, then relabel ready-for-agent.

Repoint lab (hosts/dev-new/modules/coding-lab/)

In one atomic change on dev-new:

services.lab.onecli.url = "https://onecli.cloonar.com" (REST rides TLS through web-arm's nginx vhost).
services.lab.onecli.gatewayUrl = "http://10.42.98.10:10255" (direct CONNECT proxy over wg_cloonar; nginx cannot front it).
services.lab.onecli.apiKeyFile → new sops secret on dev-new (see below); delete the local-mode auto-key-fetch oneshot (coding-lab-onecli-key) — it only worked because local mode minted keys unauthenticated.
services.lab.onecli.caFile → the central gateway's interception CA, committed to the repo as a plain PEM file (public trust material, 0644 by design; deterministic, no boot-time fetch). Fetch once from https://onecli.cloonar.com/v1/gateway/ca. If OneCLI is ever reinstalled this file must be refreshed — note that beside the file.
services.lab.onecli.dashboard = "off", drop dashboardAddr — the central dashboard has its own vhost; lab's port-mode proxy is obsolete.

Demolish the sidecar and its plumbing

Delete hosts/dev-new/modules/coding-lab/onecli.nix (both containers, the onecli podman network unit) and dev-new's networking.firewall.allowedTCPPorts = [ 8081 ].
Remove the onecli-dashboard (:8081) vhost from hosts/fw/vms/web/lab.nix.
Remove the 8081 hairpin-DNAT from hosts/fw/modules/firewall.nix (added for the old dashboard path).
The sidecar's podman volumes (onecli-pgdata, onecli-appdata) on dev-new become orphans; note in the PR that the operator can prune them — dev-new is cattle, nothing to preserve.

Secrets for the operator to add (agent must NOT edit sops files)

In hosts/dev-new/secrets.sops.yaml:

coding-lab-onecli-api-key — the key minted in the central dashboard (oc_proj_*). Module wires it with owner = lab service user, mode 0400 (apiKeyFile carries the masterKeyFile 0600-or-stricter contract).

Manual re-entry (operator, dashboard)

Grants/secrets from the sidecar do not migrate; re-enter them in the central dashboard (the sidecar is a week old — a handful of entries). Lab re-creates its per-repo agent identities lazily on first spawn.

Accepted consequences (state in the PR description)

Brief fail-closed spawn window while bento switches dev-new.
From then on, no agent run spawns while the fw↔web-arm wg_cloonar tunnel is down: the tunnel is production-critical for lab.

Verification

A spawned run gets HTTPS_PROXY pointing at 10.42.98.10:10255 and a granted service call succeeds through the gateway.
GET /api/v1/onecli/health on lab reports ok.
Dashboard reachable from LAN browser at https://onecli.cloonar.com; port 8081 answers nowhere anymore.

PR 2 of 2 — depends on #337 being merged, deployed, and manually provisioned. Do not start until the operator confirms: central OneCLI reachable at https://onecli.cloonar.com, account created (login mode), lab API key minted in the dashboard and added to dev-new sops, and the new gateway interception CA extracted. Design settled in the 2026-08-14 grill session. Keep as needs-triage until those prerequisites are done, then relabel ready-for-agent. ## Repoint lab (hosts/dev-new/modules/coding-lab/) In one atomic change on dev-new: - `services.lab.onecli.url = "https://onecli.cloonar.com"` (REST rides TLS through web-arm's nginx vhost). - `services.lab.onecli.gatewayUrl = "http://10.42.98.10:10255"` (direct CONNECT proxy over wg_cloonar; nginx cannot front it). - `services.lab.onecli.apiKeyFile` → new sops secret on dev-new (see below); delete the local-mode auto-key-fetch oneshot (`coding-lab-onecli-key`) — it only worked because local mode minted keys unauthenticated. - `services.lab.onecli.caFile` → the central gateway's interception CA, committed to the repo as a plain PEM file (public trust material, 0644 by design; deterministic, no boot-time fetch). Fetch once from `https://onecli.cloonar.com/v1/gateway/ca`. If OneCLI is ever reinstalled this file must be refreshed — note that beside the file. - `services.lab.onecli.dashboard = "off"`, drop `dashboardAddr` — the central dashboard has its own vhost; lab's port-mode proxy is obsolete. ## Demolish the sidecar and its plumbing - Delete `hosts/dev-new/modules/coding-lab/onecli.nix` (both containers, the onecli podman network unit) and dev-new's `networking.firewall.allowedTCPPorts = [ 8081 ]`. - Remove the `onecli-dashboard` (:8081) vhost from `hosts/fw/vms/web/lab.nix`. - Remove the 8081 hairpin-DNAT from `hosts/fw/modules/firewall.nix` (added for the old dashboard path). - The sidecar's podman volumes (onecli-pgdata, onecli-appdata) on dev-new become orphans; note in the PR that the operator can prune them — dev-new is cattle, nothing to preserve. ## Secrets for the operator to add (agent must NOT edit sops files) In `hosts/dev-new/secrets.sops.yaml`: - `coding-lab-onecli-api-key` — the key minted in the central dashboard (oc_proj_*). Module wires it with owner = lab service user, mode 0400 (apiKeyFile carries the masterKeyFile 0600-or-stricter contract). ## Manual re-entry (operator, dashboard) - Grants/secrets from the sidecar do not migrate; re-enter them in the central dashboard (the sidecar is a week old — a handful of entries). Lab re-creates its per-repo agent identities lazily on first spawn. ## Accepted consequences (state in the PR description) - Brief fail-closed spawn window while bento switches dev-new. - From then on, no agent run spawns while the fw↔web-arm wg_cloonar tunnel is down: the tunnel is production-critical for lab. ## Verification - A spawned run gets HTTPS_PROXY pointing at 10.42.98.10:10255 and a granted service call succeeds through the gateway. - `GET /api/v1/onecli/health` on lab reports ok. - Dashboard reachable from LAN browser at https://onecli.cloonar.com; port 8081 answers nowhere anymore.

dominik.polakovics added the

needs-triage

label

2026-08-14 12:38:50 +02:00

dominik.polakovics commented

2026-08-14 21:50:08 +02:00

Author

Owner

This was generated by AI during triage.

Agent Brief

Category: enhancement
Summary: Repoint lab on dev-new at the central OneCLI on web-arm and delete the local sidecar and all its dashboard plumbing.

Prerequisites — verified done (2026-08-14):

Central OneCLI is live at https://onecli.cloonar.com (#337 merged and deployed, ADR-0026). Login ended up as OIDC via Keycloak rather than the plain login mode this issue's body assumed — no effect on the lab side.
coding-lab-onecli-api-key is present in dev-new's sops file (commit fc57bd7).
The gateway interception CA is served unauthenticated at https://onecli.cloonar.com/v1/gateway/ca — verified fetchable from inside an agent sandbox today, so the implementing agent can fetch it once and commit the PEM itself.

Current behavior:
lab on dev-new talks to a local OneCLI sidecar (two podman containers in local mode, loopback-only), auto-fetches its API key and CA from that sidecar at boot via a oneshot, and exposes the sidecar dashboard through lab's port-mode proxy on :8081, fronted by an onecli-dashboard vhost on the web VM plus an 8081 hairpin DNAT on fw.

Desired behavior:
As specified in the issue body, in one PR:

services.lab.onecli on dev-new points at the central instance: url = "https://onecli.cloonar.com", gatewayUrl = "http://10.42.98.10:10255", apiKeyFile from the coding-lab-onecli-api-key sops secret (owner = lab service user, mode 0400), caFile = the CA PEM committed to the repo as plain 0644 trust material (with a note beside it that a OneCLI reinstall requires refreshing it).
dashboard = "off", dashboardAddr dropped — the central dashboard has its own vhost.
The sidecar and every trace of its plumbing deleted: both containers and the podman network unit, the API-key/CA fetch oneshot, dev-new's :8081 firewall opening, the web VM's onecli-dashboard vhost, and fw's 8081 hairpin DNAT.

As-built notes beyond the issue body:

The central gateway's browser-facing HTTP routes ride a separate TLS vhost on :10256 — irrelevant here; agents CONNECT to 10255 directly over wg_cloonar.
A stale, unused copy of coding-lab-onecli-api-key also sits in web-arm's sops file. Operator cleanup — the agent must NOT edit any *.sops.yaml; just mention it in the PR description.

Acceptance criteria:

A spawned run gets HTTPS_PROXY pointing at 10.42.98.10:10255 and a granted service call succeeds through the central gateway.
GET /api/v1/onecli/health on lab reports ok.
Dashboard reachable from a LAN browser at https://onecli.cloonar.com; port 8081 answers nowhere anymore (dev-new listener, web VM vhost, fw DNAT all gone).
No reference to the sidecar remains in the dev-new or fw configs.
PR description states the accepted consequences (brief fail-closed spawn window while bento switches dev-new; from then on lab spawns depend on the fw↔web-arm wg_cloonar tunnel) and notes the orphaned podman volumes on dev-new (onecli-pgdata, onecli-appdata) that the operator can prune.

Out of scope:

Any change to the central OneCLI module on web-arm.
powersync pg_hba hardening (#336).
Migrating grants/secrets out of the sidecar — the operator re-enters them in the central dashboard; lab re-creates per-repo agent identities lazily on first spawn.
Editing any sops file — the needed secret already exists on dev-new.

> *This was generated by AI during triage.* ## Agent Brief **Category:** enhancement **Summary:** Repoint lab on dev-new at the central OneCLI on web-arm and delete the local sidecar and all its dashboard plumbing. **Prerequisites — verified done (2026-08-14):** - Central OneCLI is live at `https://onecli.cloonar.com` (#337 merged and deployed, ADR-0026). Login ended up as OIDC via Keycloak rather than the plain login mode this issue's body assumed — no effect on the lab side. - `coding-lab-onecli-api-key` is present in dev-new's sops file (commit fc57bd7). - The gateway interception CA is served unauthenticated at `https://onecli.cloonar.com/v1/gateway/ca` — verified fetchable from inside an agent sandbox today, so the implementing agent can fetch it once and commit the PEM itself. **Current behavior:** lab on dev-new talks to a local OneCLI sidecar (two podman containers in local mode, loopback-only), auto-fetches its API key and CA from that sidecar at boot via a oneshot, and exposes the sidecar dashboard through lab's port-mode proxy on :8081, fronted by an `onecli-dashboard` vhost on the web VM plus an 8081 hairpin DNAT on fw. **Desired behavior:** As specified in the issue body, in one PR: - `services.lab.onecli` on dev-new points at the central instance: `url = "https://onecli.cloonar.com"`, `gatewayUrl = "http://10.42.98.10:10255"`, `apiKeyFile` from the `coding-lab-onecli-api-key` sops secret (owner = lab service user, mode 0400), `caFile` = the CA PEM committed to the repo as plain 0644 trust material (with a note beside it that a OneCLI reinstall requires refreshing it). - `dashboard = "off"`, `dashboardAddr` dropped — the central dashboard has its own vhost. - The sidecar and every trace of its plumbing deleted: both containers and the podman network unit, the API-key/CA fetch oneshot, dev-new's :8081 firewall opening, the web VM's `onecli-dashboard` vhost, and fw's 8081 hairpin DNAT. **As-built notes beyond the issue body:** - The central gateway's browser-facing HTTP routes ride a separate TLS vhost on :10256 — irrelevant here; agents CONNECT to 10255 directly over wg_cloonar. - A stale, unused copy of `coding-lab-onecli-api-key` also sits in web-arm's sops file. Operator cleanup — the agent must NOT edit any `*.sops.yaml`; just mention it in the PR description. **Acceptance criteria:** - [ ] A spawned run gets `HTTPS_PROXY` pointing at `10.42.98.10:10255` and a granted service call succeeds through the central gateway. - [ ] `GET /api/v1/onecli/health` on lab reports ok. - [ ] Dashboard reachable from a LAN browser at `https://onecli.cloonar.com`; port 8081 answers nowhere anymore (dev-new listener, web VM vhost, fw DNAT all gone). - [ ] No reference to the sidecar remains in the dev-new or fw configs. - [ ] PR description states the accepted consequences (brief fail-closed spawn window while bento switches dev-new; from then on lab spawns depend on the fw↔web-arm wg_cloonar tunnel) and notes the orphaned podman volumes on dev-new (`onecli-pgdata`, `onecli-appdata`) that the operator can prune. **Out of scope:** - Any change to the central OneCLI module on web-arm. - powersync pg_hba hardening (#336). - Migrating grants/secrets out of the sidecar — the operator re-enters them in the central dashboard; lab re-creates per-repo agent identities lazily on first spawn. - Editing any sops file — the needed secret already exists on dev-new.

dominik.polakovics added

enhancement

ready-for-agent

and removed

needs-triage

labels

2026-08-14 21:50:12 +02:00

dominik.polakovics referenced this issue from a commit

2026-08-14 22:13:51 +02:00

feat(dev-new): cut lab over to the central OneCLI, retire the sidecar

dominik.polakovics referenced this issue from a pull request that will close it,

2026-08-14 22:14:21 +02:00