feat(web-arm): central OneCLI credential gateway, WireGuard-only #339
No reviewers
Labels
No labels
bug
enhancement
in-progress
needs-info
needs-triage
p0
ready-for-agent
ready-for-human
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Cloonar/nixos!339
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "afk/337"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Stands up the central OneCLI credential gateway on web-arm: one podman container against web-arm's existing host PG14, dashboard behind an internal-only nginx vhost, and the CONNECT forward proxy published on the WireGuard address alone. This is PR 1 of 2 — repointing lab and retiring the dev-new sidecar is the follow-up.
Closes #337
What lands
hosts/web-arm/modules/onecli/default.nix— container, DB, secrets, firewall, vhost, ordering.hosts/web-arm/modules/onecli/oidc-login.patch— adds a generic OIDC login provider to the upstream web app (upstream OSS hardcodes Google as the only provider).hosts/web-arm/modules/onecli/update.sh— builds the patched image natively on web-arm, pushes it to the Forgejo registry, pins the digest here.hosts/web-arm/modules/keycloak/{default.nix,cloonar-realm.json}— declarativeonecliclient (admin-only browser flow, secret via the file vault).hosts/web-arm/configuration.nix— one import line.hosts/fw/modules/dnsmasq.nix—onecli.cloonar.com→10.42.98.10.docs/adr/0026-central-onecli-on-web-arm.md.No sops file was touched (see Operator prerequisites below).
Checkpoints from the issue
PG14 compatibility — verified, it works
Upstream's compose pins
postgres:18-alpineand upstream CI only drift-checks against 18, so PG14 is our choice, not a supported configuration. Rather than infer it, the migrations were run:prisma migrate deploy→ all 83 migrations applied, exit 0, 39 tables created.prisma migrate diff --from-url <pg14> --to-schema-datamodel) → "No difference detected", exit 0. The schema PG14 produces is identical in Prisma's model to whatschema.prismadeclares.POLICY_V2_SELECTpublished-rules query (nestedjson_agg/json_build_objectcorrelated subqueries), theINSERT ... SELECT * FROM UNNEST($1::text[], ...)telemetry batch,pg_advisory_xact_lock(hashtext(...)), and theproject_accessEXISTS lookup. All four prepared and executed cleanly.A static sweep of the migrations for PG15+ constructs (
MERGE,NULLS NOT DISTINCT,uuidv7(),ANY_VALUE,JSON_TABLE/JSON_OBJECT,random_normal,security_invoker,CREATE EXTENSION, identity columns,xid8, …) returned zero hits. The only non-trivial functions used aregen_random_uuid()(PG13+) andnum_nonnulls()(PG9.6+).So the fallback in the issue — a dedicated
postgres:18-alpinecontainer plus apg_dumppreHook — is not needed and is not implemented. Residual risk: a future upstream migration could introduce a PG15+ construct. It would fail loudly at container start (the entrypoint runsmigrate deploybefore serving), and the image is pinned by digest so it can only happen when we deliberately bump.First-account bootstrap in login mode — verified, and it is worse than "first user wins"
NEXTAUTH_SECRETbeing non-empty is the only thing that selects login mode; the entrypoint recomputes and overwritesAUTH_MODEfrom it, so settingAUTH_MODEyourself is inert. Upstream ships Google as the only OAuth provider in the OSS image; this PR carriesoidc-login.patch(see Login via our IdP below), so login runs against the Keycloak realm instead — no Google dependency anywhere.There is no admin account and no bootstrap step.
GET /v1/auth/sessionupserts the user and callsshouldBootstrapOrg(), whose OSS implementation is() => trueunconditionally — so every identity that completes the Google flow gets its own organization, project, API key and default agent. Permanently, not just the first one. The OSS edition hasrbac: falseand theInvitationmodel has no route, so there is no invite gate, no allowlist, and no admin role to hold.Operator bootstrap is therefore just: create the Keycloak client (see prerequisites), browse to the dashboard, log in via SSO. Your org is created by that first request; mint the consumer API keys (Settings → API keys) afterwards — that is how dev-new and the openclaw VM will add agents in PR 2, never via an unauthenticated IP carve-out (which could not distinguish a host's lab daemon from the agents NATed behind the same address).
The access control is ours, not OneCLI's. Two layers, both in this PR or adjacent to it:
10.42.96.0/24,10.42.97.0/24,10.42.98.0/24and denies everything else — the dashboard is unreachable from the internet at all.onecliclient is bound to the realm'sbrowser-spnego admin-onlyflow (as Grafana is), so only admin-role identities can complete a login at all; anyone else is cut off at the IdP before OneCLI ever sees them. That matters because OneCLI has no invite gate of its own and every successful login gets a working credential vault.If
NEXTAUTH_SECRETis set without a configured provider, the app hard-redirects to/setup-error?code=oauth-misconfigured— it fails closed rather than silently dropping to local mode. The patch extends theoauthConfiguredcheck to countOIDC_ISSUER, so that property is preserved.At-rest encryption key — located, and moved out of
/app/dataThe key is
SECRET_ENCRYPTION_KEY: base64 of exactly 32 raw bytes, AES-256-GCM, wire format{iv}:{authTag}:{ciphertext}, implemented compatibly in bothpackages/api/src/lib/crypto.ts(Node) andapps/gateway/src/crypto.rs(Rust). It is not derived fromNEXTAUTH_SECRET.If unset,
docker/entrypoint.shgenerates one and persists it to/app/data/secret-encryption-key. This module sets it explicitly from sops instead, which collapses the restore surface to database dump + sops — the shape the rest of this repo already uses — instead of a file living inside a bind mount./app/datathen holds only: the gateway's generated MITM CA (gateway/ca.key,ca.pem) andruntime-config.json, which is rewritten on every start. It is still bind-mounted to/var/lib/onecliand still backed up, so a restore is transparent to agents that pinned the CA — but nothing there is needed to decrypt stored credentials.fw forwarding to 10.42.98.10 — already in place, no change needed
hosts/fw/modules/firewall.nix:137already acceptsiifname { "lan", "server", "vserver", "wg_cloonar" } oifname { …, "wg_cloonar", … }, and web-arm is already a provisioned peer withallowedIPs = [ 10.42.98.10/32 ]. Return traffic rides the generic established/related rule. No new forward rule, and no DNAT — traffic reaching10.42.98.10over the tunnel only needs routing, which the interface's own10.42.98.1/24provides.One consequence worth knowing: postrouting masquerades traffic leaving via
wg_cloonar(firewall.nix:199), so LAN and server-VLAN clients arrive at web-arm as10.42.98.1. That is inside10.42.98.0/24, so the vhost allowlist still passes — but all internal clients collapse to one apparent source IP, and the.96/.97allow lines are belt-and-braces rather than load-bearing.Restore drill — NOT DONE, and it is the one thing still open
The issue makes a restore drill the definition of done, and I could not perform it: this run has no SSH access to web-arm (host key verification fails), and the drill needs a running instance with a stored secret. A credential store with an untested restore is not backed up — please run this before trusting it.
The drill, concretely:
podman stop onecli; take/var/backup/postgresql/onecli.sql.gzand a copy of/var/lib/onecli.oneclidatabase, restore the dump, restore/var/lib/onecli, start a fresh container with the sameSECRET_ENCRYPTION_KEYfrom sops.Login via our IdP — the OIDC patch
Upstream's OSS web app hardcodes Google as its sole NextAuth provider (
apps/web/src/lib/auth/nextauth-config.ts), so using Keycloak requires a source patch and therefore our own image build.oidc-login.patch(10 files, +86/−20) addsOIDC_ISSUER/OIDC_CLIENT_ID/OIDC_CLIENT_SECRET, registers a generic OIDC provider (idoidc, via next-auth's Keycloak template — plain OIDC discovery, nothing Keycloak-specific) that takes precedence over Google when set, threads the provider id from the entrypoint-written runtime config through to the login button andsignIn()call, and countsOIDC_ISSUERin the entrypoint'soauthConfigured. Google support is untouched whenOIDC_ISSUERis unset, which keeps the patch upstreamable — an upstream PR is the exit ramp that retires the fork.Verified: the patch applies cleanly to the pristine v1.45.0 tree, and the patched web app compiles —
pnpm install --frozen-lockfile,prisma generate,pnpm build --filter=@onecli/weball pass (Next.js build includes the type check). The gateway needs no change: its session validation just verifies theNEXTAUTH_SECRET-signed JWT, provider-agnostic, and both halves ship in one image so they cannot drift apart.Maintenance shape: per upstream release, run
update.sh <tag>(rebuild + digest pin); if upstream refactored the auth wiring, the patch fails to apply at build time and needs a rebase first. Failures are loud, never silent.Operator prerequisites — four sops keys, a Keycloak client, and the first image build
The module consumes four secrets that do not exist yet. Until they are added, sops-nix activation will fail on web-arm (evaluation is unaffected, so the dry-build passes either way).
onecli-db-passwordonecliPG role — keep it alphanumeric, it is interpolated into apostgresql://URL and@ : / ? #would break parsingonecli-nextauth-secretopenssl rand -hex 32onecli-secret-encryption-keyopenssl rand -base64 32. The container refuses any other length. Losing this makes every stored credential permanently undecryptableonecli-oidc-client-secretonecliKeycloak clientKeycloak client — declarative, nothing to click.
cloonar-realm.jsonnow carries theonecliclient (confidential, standard flow only, redirect URIhttps://onecli.cloonar.com/api/auth/callback/oidc), converged by keycloak-config-cli. Its secret rides Keycloak's file vault from the sameonecli-oidc-client-secretsops key the OneCLI container consumes — the grafana pattern, one key feeding both sides — and the browser flow is bound tobrowser-spnego admin-only. Generate the value withopenssl rand -hex 32, add it under that one key, and both consumers pick it up on the next switch.First image build (before merge):
podman login git.cloonar.comwith an account that may write packages under theinfrastructureorg.hosts/web-arm/modules/onecli/update.sh v1.45.0from your checkout — it builds the patched image natively on web-arm, pushesgit.cloonar.com/infrastructure/onecli:1.45.0-oidc1, and seds the digest intodefault.nix. Commit that digest bump onto this branch; the image reference is tag-only until then, deliberately, since the digest cannot exist before the first push.infrastructure/oneclipackage public in Forgejo (or web-arm's pull will need registry credentials the module does not provision).One thing the issue did not anticipate: the gateway needs TLS too
The issue treats 10255 as purely agent-facing. It is not — the dashboard's own JavaScript fetches the gateway's HTTP routes from the browser, via
getGatewayApiUrl()(apps/web/src/hooks/use-vault-status.ts), which prefers theGATEWAY_API_URLvalue injected aswindow.__GATEWAY_API_URL__. Call sites:lib/api/approvals.ts,lib/api/cache.ts,hooks/use-vault-status.ts,lib/api/onepassword.ts,hooks/use-onepassword.ts.So a plain
http://…:10255value would be mixed-content-blocked on an https dashboard, and it fails silently:<ApprovalsBell />is mounted on every dashboard page and, in the OSS edition, polls unconditionally (orgScopedUIisfalse, sohasProjectContextis always true). The error branch just backs off to 30s, so the bell shows permanently empty and approve/deny is impossible — any rule carryingrequire_approvalwould hold agent requests until they auto-deny atexpiresAt. This is the headline 1.45.0 feature.fetchStatusswallows the error and reports "not connected".apiFetchis same-origin/relative in OSS, so agents, secrets, connections, grants, health and the install page all ride the main vhost.Leaving
GATEWAY_API_URLat its defaulthttp://localhost:10255is not a fix either —localhostis a trustworthy origin so it is not blocked, but in the browser it means the operator's own laptop, so the same features fail with connection-refused instead.The fix here: a second nginx server block on the same hostname, port 10256, TLS via
useACMEHoston the existing cert, proxying to the gateway's loopback publish, with the same internal-subnet allowlist and a 3600s read timeout for the approvals long-poll.GATEWAY_API_URL = https://onecli.cloonar.com:10256.Same hostname rather than a second name is load-bearing: the NextAuth cookie is host-only (no
Domainattribute), soonecli-gw.cloonar.comwould not receive it and thesecredentials: "include"fetches would 401. Cookies ignore the port, so a second TLS port on the same name works — which is exactly the topology upstream's own compose has (127.0.0.1:10254and127.0.0.1:10255). CONNECT never passes through nginx: the gateway dispatches on method, and agents still dial10255directly over the tunnel.GATEWAY_BASE_URL = onecli.cloonar.com:10255(no scheme) is confirmed correct — it is interpolated afterhttp://x:${agent.accessToken}@inpackages/api/src/routes/container-config.ts:187, so a scheme there would silently produce a broken proxy URL. Note the credential is the per-agent access token, not the project API key.Known upstream defect, not fixable from here: the install page renders
onecli config set api-host http://localhost:10255, becauseNEXT_PUBLIC_API_URLis baked at build time fromAPI_DOMAIN, which the published image never sets. No runtime env var corrects it — settingAPI_DOMAINin the podman unit does nothing. Users must substitute the real host by hand.Deployment shape
git.cloonar.com/infrastructure/onecli:1.45.0-oidc1— our own build of the upstreamv1.45.0source tag withoidc-login.patchapplied, built natively on web-arm (the fleet's only aarch64 host, so no emulation) byupdate.shand pushed to the Forgejo registry.update.shpins the digest after the first push (see prerequisites), restoring the powersync tag-and-digest property./v1) published to127.0.0.1only, fronted byonecli.cloonar.comwith the internal-subnet allowlist. TLS comes from the existing global lego DNS-01 defaults — no per-domain ACME block needed.10.42.98.10for agents, plus127.0.0.1for nginx. It is a CONNECT forward proxy, so the proxying itself cannot ride nginx; consumers dial10.42.98.10:10255directly over the tunnel. No public bind, no new WAN opening.wg_cloonaronly.v6egressbridge gateway10.89.0.1, with a scram-sha-256pg_hbaline scoped to db+role+subnet — deliberately not powersync'strust, because this database holds the grant definitions.ensureUserscannot set a password, so anALTER USERoneshot does it (the fw/piped pattern), ordered before the container./app/databind-mounted to/var/lib/onecli(0700, uid/gid 1000 = the image'snodeuser) rather than a named volume, because the borg job excludes/var/lib/containers/.services.postgresqlBackup.databasesgainsonecli, so the nightly dump rides the existing borg job.wireguard-wg_cloonar.service: the10.42.98.10port publish fails outright while the interface is down.ADR-0012's warning, now realised
Joining
v6egressmakes OneCLI the "second container" ADR-0012 named. It shares the subnet with powersync'strustpg_hba rule, so a compromise of the OneCLI container gains write access topowersync_storage. OneCLI's own database is not exposed the same way — its rule requires a password. The hardening is #336 and stays out of scope here; ADR-0026 records the exposure rather than leaving it implicit.v6egresswas chosen over the default bridge deliberately: the gateway proxies agent traffic to arbitrary upstreams, some of which are v6-only, which is exactly what ADR-0011 exists for.Verification
:: web-arm OK) — the real gate, run on the first commit.nix-instantiateon web-arm's full config, reading values back rather than trusting the source: image, all three port publishes,GATEWAY_API_URL/GATEWAY_BASE_URL, the rendered env-file path, the mergedpostgresqlBackup.databases, the tmpfiles rule, the mergedwg_cloonarport list, and both vhosts (serverName,useACMEHost,listen,proxyPass).config.assertionshas no failures.nixpkgs-fmt --checkclean on all three changed.nixfiles.fw's dry-build could not be run in this environment, and the hook was bypassed for that one commit.
scripts/test-configuration fwfails aterror: Failed to fetch git repository 'https://github.com/astro/microvm.nix'— this sandbox's egress proxy cannot carry git-over-https, sofw'smicrovm.nixinput is unfetchable here. That is environmental and unrelated to the change (a dnsmasqaddress=entry cannot cause a git fetch to fail). In its place I evaluated the dnsmasq module standalone and confirmed the entry renders as"/onecli.cloonar.com/10.42.98.10". Worth a second pair of eyes on that one line, since it did not go through the usual gate.Everything else — the cert, the OAuth round trip, the container→PG hop, the gateway bind, the nginx config's own
nginx -t(which only runs at build/deploy, not at instantiate), and the restore drill — is only provable on the deployed host.[autoland] verdict: pass
Verdict: CONCERNS (non-blocking on the diff; one hard operator precondition on the merge itself).
Signal relied on:
labctl pr checksreportsstate: none— no CI vouches for this repo — so the project's own gate was run here on the merge result (afk/337 + origin/main, which merges cleanly):scripts/test-configuration web-arm→ exit 0 (Dry-build for host 'web-arm' completed successfully).nixpkgs-fmt --checkon all three changed.nixfiles → 0 of 3 would be reformatted.scripts/test-configuration fw→ fails, atFailed to fetch git repository 'https://github.com/astro/microvm.nix'. Confirmed environmental by a control run: unmodifiedorigin/mainfails identically at the same input. The fw change is one dnsmasqaddress=line;networkPrefix = "10.42"(hosts/fw/configuration.nix:90) renders it/onecli.cloonar.com/10.42.98.10, matching its siblings.Diff scope matches #337 exactly — 4 files, no drive-by changes. ADR-0026 is the next free number. CONTEXT.md keeps no ADR index, so no update is owed.
Verified beyond the author's own account:
services.postgresql.authenticationmerges tohost onecli onecli 10.89.0.0/24 scram-sha-256above powersync'shost powersync_storage powersync_storage 10.89.0.0/24 trust. Both are db+role-scoped, so pg_hba first-match cannot let powersync'strustshadow the onecli rule.127.0.0.1:10254,10.42.98.10:10255,127.0.0.1:10255), the four env values,postgresqlBackup.databases = [grafana onecli powersync_storage keycloak],wg_cloonarTCP[3306 10255 10256].enableACME = true; acmeRoot = null;is the established web-arm pattern (updns, victoriametrics, nextcloud, loki).CONCERN 1 —
onecli-db-password.servicecan lose a first-activation racehosts/web-arm/modules/onecli/default.nix:60 orders the oneshot
after = [ "postgresql.service" ]. But in nixos-26.05 theoneclirole is created bypostgresql-setup.service, a separate unit that is itself onlyafter/requirespostgresql.service(nixpkgsnixos/modules/services/databases/postgresql.nix:871);systemd.targets.postgresqlis what requires both. Evaluated ordering confirms there is no edge betweenonecli-db-password.serviceandpostgresql-setup.service— both merely followpostgresql.service, so systemd may run them concurrently.If the oneshot wins that race on the first activation,
ALTER USER onecli WITH PASSWORDfails withrole "onecli" does not exist. There is noRestart=, so the unit stays failed, andpodman-onecli(whichrequiresit) never starts. web-arm'spostgresql-setupalso carries anExecStartPostrunning immich's pgvectors SQL, so it is not instantaneous.Narrow but real: only the first activation is exposed — once the role exists it persists in PGDATA, so later boots succeed regardless of order. Recovery is
systemctl restart onecli-db-password podman-onecli. One-line fix: order afterpostgresql.target(the idiom nixpkgs provides) or explicitly afterpostgresql-setup.service.CONCERN 2 — the five sops keys are absent; merging before they exist breaks web-arm's deploy loop
Verified directly:
hosts/web-arm/secrets.sops.yamlat this head contains zero occurrences ofonecli(sops leaves keys in plaintext, so this is checkable without decrypting). The PR discloses that activation fails until they are added; the consequence worth stating explicitly is that bento switches web-arm every ~5 min, so merging first leaves every subsequent web-arm config change failing to apply until the keys land.Order of operations: add all five keys, then merge.
CONCERN 3 — the restore drill remains undone
#337 makes it the definition of done. It is inherently post-deploy (it needs a running instance with a stored secret), so it cannot gate the merge — but it does gate trusting the store. The step that matters most is the author's step 5: restoring with a deliberately wrong
SECRET_ENCRYPTION_KEYand confirming it fails closed.Not merged — awaiting the human's explicit confirmation.
Pushed
5b7ee48toafk/337, fixing CONCERN 1 from the validation above.postgresql.targetcarriesRequires=postgresql.service postgresql-setup.service, and a target with defaultDefaultDependencies=implicitly gainsAfter=on itsRequires=members (systemd.target(5)) — so the target is only reached once the role exists. This is also the nixpkgs convention (cf.services/mail/roundcube.nix:270, which uses the samerequires+afterpair).Re-verified on the resolution commit, since no gate had seen it:
after = ["postgresql.target"],requires = ["postgresql.target"], and the target'sRequires=postgresql.service postgresql-setup.servicescripts/test-configuration web-arm→ exit 0nixpkgs-fmt --check→ cleanpodman-onecliwas deliberately left alone: itrequiresthis oneshot, so it inherits the corrected ordering transitively.CONCERN 2 (five sops keys absent) and CONCERN 3 (restore drill) are unchanged and still stand.
Auth reworked on this branch (
ad8fcab,2d8a903): Google is gone entirely. Login now runs against the Keycloak realm viaoidc-login.patch(generic OIDC provider; patched web app verified to build+typecheck at v1.45.0), image becomes our own buildgit.cloonar.com/infrastructure/onecli:1.45.0-oidc1produced by the newupdate.shon web-arm. PR body updated in place — see Login via our IdP and the reworked operator prerequisites: now four sops keys (onecli-nextauth-secretandonecli-oidc-client-secretstill missing; the google pair is no longer needed), plus the Keycloak client and the first image build before merge.