labctl breaks behind an SSO/auth proxy — session LAB_URL follows --base-url out through Authelia #30

New issue

Closed

opened 2026-07-07 22:19:35 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-07-07 22:19:35 +02:00

Owner

This was generated by AI during triage.

Summary

labctl fails entirely when lab runs behind an SSO/auth proxy (nginx + Authelia header-auth, as in the lab.cloonar.com production deploy). Every labctl call inside an agent session gets a 302 to the SSO login page; labctl follows the redirect, receives the HTML login page, and tries to parse it as JSON — failing with invalid character '<' looking for beginning of value. The run's only tracker surface (D10) is dead in that deployment.

Current behavior (confirmed by code trace)

The operator sets --base-url (LAB_BASE_URL) to the external HTTPS origin — required for Secure cookies and the CSRF Origin check.
The LAB_URL handed to every spawned session is derived from --base-url: the labURL() helper in the lab command returns the base URL verbatim when it is set. So sessions get LAB_URL=https://lab.cloonar.com (the public FQDN).
labctl issue list therefore issues GET https://lab.cloonar.com/agent/v1/issues with an Authorization: Bearer lab_run_… header. That request leaves the lab host and hairpins out to the public reverse proxy.
The reverse proxy gates every path (including /agent/v1/) through Authelia (auth_request). Authelia has no concept of lab run tokens, sees no SSO session cookie, and returns 401 → the proxy maps that to a 302 to the Authelia login portal.
labctl's HTTP client is a bare http.Client with no CheckRedirect, so it follows the 302, gets 200 OK + an HTML login page, and json.Unmarshal chokes on the leading <.

Root cause

--base-url does double duty: it is both (a) the browser-facing origin (must be the external https:// URL) and (b) the source of LAB_URL for machine-to-machine traffic (which should never touch the human SSO gateway). Behind an auth proxy these two roles conflict.

The agent API (/agent/v1) is designed to self-authenticate with run tokens only — no cookies, no CSRF — and is deliberately mounted outside the operator auth/CSRF middleware. Routing it through Authelia is therefore wrong by construction: machine traffic should stay on loopback and never reach the proxy.

Chosen fix (approach A + C — decided with the maintainer)

Decouple the session-facing URL from --base-url and default machine traffic to loopback, plus harden labctl so a proxied misconfig fails loudly instead of parsing HTML as JSON.

Explicitly rejected alternative: an nginx carve-out that exposes /agent/v1/ unauthenticated on the public vhost. It works but hairpins traffic and widens the public attack surface; loopback is strictly better.

Agent Brief

Category: bug
Summary: Give lab a dedicated session-facing URL so labctl talks to the server directly (loopback by default) instead of being forced out through the external --base-url and its SSO/auth proxy; harden labctl to fail clearly on redirects.

Current behavior:
The LAB_URL value injected into every spawned agent session is derived from --base-url: when --base-url is set, sessions receive that external URL. Behind an authenticating reverse proxy this routes labctl's run-token-authenticated /agent/v1 calls through the SSO gateway, which 302-redirects them to a login page. labctl's HTTP client follows redirects and then fails to decode the HTML login page as JSON, so every labctl command breaks.

Desired behavior:

lab exposes a dedicated knob for the session-facing base URL, independent of --base-url. Precedence for the value handed to sessions as LAB_URL becomes: the new agent-URL knob if set, else --base-url if set, else http://127.0.0.1:<listen-port> (unchanged fallback).
The NixOS module defaults this knob to a loopback URL so the out-of-the-box behind-a-proxy deployment keeps machine traffic on loopback and never touches the auth proxy.
labctl no longer silently follows redirects. A 3xx response (or a non-JSON body/content-type) from the agent API surfaces as an actionable error that names the redirect target and hints that LAB_URL may be pointed at an auth proxy — instead of invalid character '<'.
--base-url keeps its existing role for Secure-cookie detection and the CSRF Origin check, unchanged.

Key interfaces:

A new server flag + env var, e.g. --agent-url / LAB_AGENT_URL, resolved in the config parser with the standard flag > env > default precedence and validated as an absolute http(s) URL exactly like --base-url (reject relative/other-scheme values at startup with a clear message). Surface it on the Config struct.
The helper that computes the session LAB_URL (currently labURL() in the lab command): change its precedence to agent-URL → base-URL → http://127.0.0.1:<port>.
The labctl HTTP Client (its do method): set http.Client.CheckRedirect to return http.ErrUseLastResponse, and when the response status is a redirect (or the payload clearly isn't the JSON error/OK envelope), return an error that includes the Location and the "is LAB_URL behind an auth proxy?" hint.
The NixOS module services.lab: add an agentUrl option (nullable string). Default it to a loopback URL derived from the listen port already computed in the module (http://127.0.0.1:<listenPort>), and pass it through as --agent-url. Document that it exists so machine traffic bypasses any front proxy; the operator can override it if sessions run off-host.

Acceptance criteria:

lab accepts --agent-url / LAB_AGENT_URL; an invalid value (relative or non-http(s)) is rejected at startup with a clear error, mirroring --base-url validation.
With the agent-URL knob set, spawned sessions receive that exact value as LAB_URL. With it unset, behavior is unchanged: --base-url when set, else http://127.0.0.1:<port>.
labctl does not follow HTTP redirects; a 3xx from the agent API produces an actionable error mentioning the redirect Location and that LAB_URL may be behind an auth proxy, rather than a JSON-decode error on HTML.
The NixOS module exposes services.lab.agentUrl, defaulting to a loopback URL derived from listenAddr, and passes --agent-url to the unit — so a proxy-fronted deployment keeps labctl traffic on loopback with no extra operator config.
Unit tests cover: config precedence + validation for the agent-URL knob; the session-URL precedence (agent-URL > base-URL > loopback); and labctl's redirect/non-JSON handling.
nix flake check (go test + golangci-lint) passes.

Out of scope:

The cloonar-nixos deployment change (bumping the pinned rev in utils/modules/coding-lab/default.nix, and optionally setting services.lab.agentUrl). That lives in the cloonar-nixos repo and is a follow-up once this change lands and is available at a rev.
The rejected nginx carve-out (exposing /agent/v1/ unauthenticated on the public vhost).
Any change to the agent API's run-token authentication, or to --base-url's role in Secure-cookie / CSRF handling.

> *This was generated by AI during triage.* ## Summary `labctl` fails entirely when lab runs behind an SSO/auth proxy (nginx + Authelia header-auth, as in the `lab.cloonar.com` production deploy). Every `labctl` call inside an agent session gets a 302 to the SSO login page; labctl follows the redirect, receives the HTML login page, and tries to parse it as JSON — failing with `invalid character '<' looking for beginning of value`. The run's only tracker surface (D10) is dead in that deployment. ## Current behavior (confirmed by code trace) 1. The operator sets `--base-url` (`LAB_BASE_URL`) to the external HTTPS origin — required for Secure cookies and the CSRF Origin check. 2. The `LAB_URL` handed to every spawned session is **derived from `--base-url`**: the `labURL()` helper in the `lab` command returns the base URL verbatim when it is set. So sessions get `LAB_URL=https://lab.cloonar.com` (the public FQDN). 3. `labctl issue list` therefore issues `GET https://lab.cloonar.com/agent/v1/issues` with an `Authorization: Bearer lab_run_…` header. That request leaves the lab host and hairpins out to the public reverse proxy. 4. The reverse proxy gates **every** path (including `/agent/v1/`) through Authelia (`auth_request`). Authelia has no concept of lab run tokens, sees no SSO session cookie, and returns 401 → the proxy maps that to a **302** to the Authelia login portal. 5. labctl's HTTP client is a bare `http.Client` with **no `CheckRedirect`**, so it follows the 302, gets `200 OK` + an HTML login page, and `json.Unmarshal` chokes on the leading `<`. ## Root cause `--base-url` does double duty: it is both (a) the browser-facing origin (must be the external `https://` URL) and (b) the source of `LAB_URL` for **machine-to-machine** traffic (which should never touch the human SSO gateway). Behind an auth proxy these two roles conflict. The agent API (`/agent/v1`) is designed to self-authenticate with run tokens only — no cookies, no CSRF — and is deliberately mounted **outside** the operator auth/CSRF middleware. Routing it through Authelia is therefore wrong by construction: machine traffic should stay on loopback and never reach the proxy. ## Chosen fix (approach A + C — decided with the maintainer) Decouple the session-facing URL from `--base-url` and default machine traffic to loopback, plus harden labctl so a proxied misconfig fails loudly instead of parsing HTML as JSON. Explicitly **rejected** alternative: an nginx carve-out that exposes `/agent/v1/` unauthenticated on the public vhost. It works but hairpins traffic and widens the public attack surface; loopback is strictly better. --- ## Agent Brief **Category:** bug **Summary:** Give lab a dedicated session-facing URL so `labctl` talks to the server directly (loopback by default) instead of being forced out through the external `--base-url` and its SSO/auth proxy; harden labctl to fail clearly on redirects. **Current behavior:** The `LAB_URL` value injected into every spawned agent session is derived from `--base-url`: when `--base-url` is set, sessions receive that external URL. Behind an authenticating reverse proxy this routes labctl's run-token-authenticated `/agent/v1` calls through the SSO gateway, which 302-redirects them to a login page. labctl's HTTP client follows redirects and then fails to decode the HTML login page as JSON, so every labctl command breaks. **Desired behavior:** - lab exposes a dedicated knob for the session-facing base URL, independent of `--base-url`. Precedence for the value handed to sessions as `LAB_URL` becomes: the new agent-URL knob if set, else `--base-url` if set, else `http://127.0.0.1:<listen-port>` (unchanged fallback). - The NixOS module defaults this knob to a loopback URL so the out-of-the-box behind-a-proxy deployment keeps machine traffic on loopback and never touches the auth proxy. - labctl no longer silently follows redirects. A 3xx response (or a non-JSON body/content-type) from the agent API surfaces as an actionable error that names the redirect target and hints that `LAB_URL` may be pointed at an auth proxy — instead of `invalid character '<'`. - `--base-url` keeps its existing role for Secure-cookie detection and the CSRF Origin check, unchanged. **Key interfaces:** - A new server flag + env var, e.g. `--agent-url` / `LAB_AGENT_URL`, resolved in the config parser with the standard flag > env > default precedence and validated as an absolute `http(s)` URL exactly like `--base-url` (reject relative/other-scheme values at startup with a clear message). Surface it on the `Config` struct. - The helper that computes the session `LAB_URL` (currently `labURL()` in the `lab` command): change its precedence to agent-URL → base-URL → `http://127.0.0.1:<port>`. - The labctl HTTP `Client` (its `do` method): set `http.Client.CheckRedirect` to return `http.ErrUseLastResponse`, and when the response status is a redirect (or the payload clearly isn't the JSON error/OK envelope), return an error that includes the `Location` and the "is LAB_URL behind an auth proxy?" hint. - The NixOS module `services.lab`: add an `agentUrl` option (nullable string). Default it to a loopback URL derived from the listen port already computed in the module (`http://127.0.0.1:<listenPort>`), and pass it through as `--agent-url`. Document that it exists so machine traffic bypasses any front proxy; the operator can override it if sessions run off-host. **Acceptance criteria:** - [ ] `lab` accepts `--agent-url` / `LAB_AGENT_URL`; an invalid value (relative or non-http(s)) is rejected at startup with a clear error, mirroring `--base-url` validation. - [ ] With the agent-URL knob set, spawned sessions receive that exact value as `LAB_URL`. With it unset, behavior is unchanged: `--base-url` when set, else `http://127.0.0.1:<port>`. - [ ] labctl does not follow HTTP redirects; a 3xx from the agent API produces an actionable error mentioning the redirect `Location` and that `LAB_URL` may be behind an auth proxy, rather than a JSON-decode error on HTML. - [ ] The NixOS module exposes `services.lab.agentUrl`, defaulting to a loopback URL derived from `listenAddr`, and passes `--agent-url` to the unit — so a proxy-fronted deployment keeps `labctl` traffic on loopback with no extra operator config. - [ ] Unit tests cover: config precedence + validation for the agent-URL knob; the session-URL precedence (agent-URL > base-URL > loopback); and labctl's redirect/non-JSON handling. - [ ] `nix flake check` (go test + golangci-lint) passes. **Out of scope:** - The cloonar-nixos deployment change (bumping the pinned `rev` in `utils/modules/coding-lab/default.nix`, and optionally setting `services.lab.agentUrl`). That lives in the cloonar-nixos repo and is a follow-up once this change lands and is available at a rev. - The rejected nginx carve-out (exposing `/agent/v1/` unauthenticated on the public vhost). - Any change to the agent API's run-token authentication, or to `--base-url`'s role in Secure-cookie / CSRF handling.

dominik.polakovics added the

ready-for-agent

bug

labels

2026-07-07 22:19:35 +02:00

dominik.polakovics referenced this issue from a commit

2026-07-07 22:54:33 +02:00

fix(agent-url): decouple session LAB_URL from --base-url; harden labctl vs auth proxies

dominik.polakovics referenced this issue from a pull request that will close it,

2026-07-07 22:54:55 +02:00

fix(agent-url): decouple session LAB_URL from --base-url; harden labctl vs auth proxies (#30) #31

dominik.polakovics referenced this issue