Cloonar/coding-lab

Fork 0

Server-owned credential authority: per-run OAuth snapshots fork the token family, rotation logs everyone out (grilled 2026-07-24) #222

New issue

Closed

opened 2026-07-24 22:34:43 +02:00 by dominik.polakovics · 1 comment

dominik.polakovics commented

2026-07-24 22:34:43 +02:00

Owner

This was generated by AI during triage.

Symptom

Since the container-isolation slices landed, the operator repeatedly hits "not logged in" — in panes, and as the host-level login banner / spawn refusals. Re-login fixes it for a few hours, then it recurs.

Root cause (diagnosed 2026-07-24 on the dev host)

InjectCredentials (#202, internal/provider/claudecode/inject.go) snapshot-copies the master .credentials.json into every run's private instance HOME, with deliberately no write-back. Anthropic's OAuth rotates refresh tokens: when a long-lived instance's ~8h access token expires, its CLI refreshes its private copy, receives a new token family, and invalidates the refresh token that the master store and every other snapshot still hold. When the master's access token next expires, the host-side refresh fails → claude auth status reads logged-out → spawn gate refuses, new snapshots are dead on arrival. Re-login mints a fresh family and the cycle restarts.

Live evidence (token sha256-prefix fingerprints, not secrets):

holder	spawned	creds last written	family
master `/var/lib/lab/.claude/.credentials.json`	—	07-24 22:19 (re-login)	`c6822a01/6f9f9848`
`run_a9c8…`	07-23 15:06	07-24 21:06 (self-refresh)	`968f977e/571a3fb5`
`run_68753…`	07-23 21:44	07-24 21:44 (self-refresh)	`64ff94a7/13d3372b`
`run_11e5…`	07-24 21:13	21:13 (spawn copy)	stale earlier family

Three independently-refreshed families alive at once, each rotation orphaning all other holders. Pre-#202 every session shared one host credentials file, so exactly one family ever existed — that's why this only appeared with the containerization work.

Side effect: wipe-on-stop can shred the only valid family if the newest rotation happened inside an instance — forced re-login even with nothing else running.

codex almost certainly carries the same latent bug: same snapshot design (internal/provider/codex/inject.go), and OpenAI also rotates refresh tokens.

Settled design (grilled 2026-07-24)

Invariant: one refresher per provider grant; instances are consumers, never refreshers. Lab owns the grant without speaking any provider's OAuth protocol. Five decisions from the grilling session:

Refresh clock — blind periodic poke. A timer periodically runs the provider's refresh-trigger command on the host; the provider's own CLI decides whether the token is near expiry and rotates. Lab detects rotation purely by file change (opaque signature over the credential files), never parsing token contents — no new coupling to any provider's token schema.
Fan-out on rotation — immediate. When the master's signature changes, re-inject into every live instance HOME through the existing per-provider InjectCredentials seam (single call site by design; instance HOMEs are host paths even for container runs). Probe evidence below shows rotation invalidates the old family's access tokens within minutes, so fan-out must directly follow any detected master change. Copies kept fresh never near expiry → instances never self-refresh → no thundering herd.
Adopt-back — eagerly on detection. If an instance self-refreshes anyway (safety net), the scan detects an instance signature that differs from what lab last injected there and adopts that family into the master immediately, then fans out — waiting invites a second instance to fork yet another family. Tie-break when several instances changed in one scan: newest credential-file mtime wins.
Wipe races — adopt-check before every wipe, signature persisted. The last-injected signature is persisted per run. The stop path synchronously compares signatures → adopts a self-refreshed family → then wipes; startup sweep does the same per orphan home before wiping it. Closes both the stop race and the restart-after-downtime race where an orphan home holds the only valid family.
Seam shape — three new methods on provider.AgentProvider:
- RefreshCredentials(ctx) — drive the provider's CLI to refresh the master store;
- CredentialsSig(home) — opaque change-detector over the credential file(s) under a home, "" meaning the master store (the SpoolSig precedent);
- AdoptCredentials(instanceHome) — copy an instance's credentials back into the master store.
  Core owns the rotation loop; adapters own every file path and CLI recipe. Fan-out reuses InjectCredentials.

Rejected alternatives, for the record: shared credentials file (claude derives the creds path from CLAUDE_CONFIG_DIR, which must stay per-run; tmp+rename replaces any symlink/hardlink on first refresh — the agent.sock inode lesson of ADR-0052 — and it reopens the isolation seam #202 closed); sync-back alone (all holders share one expiry clock, so the concurrent-refresh herd at T+8h is the common case); adopt-back-only-on-master-failure (leaves a window up to the remaining master token lifetime where the fleet holds a doomed family).

Compat probe results (claude, probed live 2026-07-24)

A real API call is a confirmed non-interactive refresh trigger. With the master's expiresAt forged 60 s into the past, claude -p --model haiku (under CLAUDE_CONFIG_DIR pointed at the master) succeeded and rewrote the master store in place (23:12:42, 509 → 945 bytes) with a fresh valid family — the next spawn ran on it with no re-login.
The CLI reads credential state from disk before refreshing — it honored the forged on-disk expiresAt rather than any in-memory view.
claude auth status does not rewrite the credentials file when the token is valid (clean control run). Whether it refreshes an expired token was not tested — moot given the confirmed trigger above.
Rotation invalidates the previous family fast. A running session still holding the pre-rotation family — its access token hours from expiry — hit "Login expired" within minutes of the master rotation. Consequence for the design: fan-out must directly follow rotation, and an instance mid-request in that window eats one failed request before its re-injected copy takes over.
Grace-window probe dropped by decision: deliberately double-spending a refresh token risks the operator's live grant, and eager adopt-back removed the design's dependency on the answer.

Remaining probes (codex — fold into implementation, not blocking)

What non-interactively triggers a refresh for codex (the codex exec analogue of claude -p)?
Does codex read auth state from disk before refreshing?

References

#202 per-run instance HOME with credential copy/wipe (introduced the snapshot design; its comment already anticipates this fix: "a server-side credential proxy can later replace this copy wholesale")
ADR-0052 container runner (mount inventory; why instance HOMEs are host-writable)
internal/provider/claudecode/inject.go, internal/provider/claudecode/auth.go, internal/provider/codex/inject.go

> *This was generated by AI during triage.* ## Symptom Since the container-isolation slices landed, the operator repeatedly hits "not logged in" — in panes, and as the host-level login banner / spawn refusals. Re-login fixes it for a few hours, then it recurs. ## Root cause (diagnosed 2026-07-24 on the dev host) `InjectCredentials` (#202, `internal/provider/claudecode/inject.go`) snapshot-copies the master `.credentials.json` into every run's private instance HOME, with deliberately no write-back. Anthropic's OAuth rotates refresh tokens: when a long-lived instance's ~8h access token expires, its CLI refreshes **its private copy**, receives a new token family, and invalidates the refresh token that the master store and every other snapshot still hold. When the master's access token next expires, the host-side refresh fails → `claude auth status` reads logged-out → spawn gate refuses, new snapshots are dead on arrival. Re-login mints a fresh family and the cycle restarts. Live evidence (token sha256-prefix fingerprints, not secrets): | holder | spawned | creds last written | family | |---|---|---|---| | master `/var/lib/lab/.claude/.credentials.json` | — | 07-24 22:19 (re-login) | `c6822a01/6f9f9848` | | `run_a9c8…` | 07-23 15:06 | **07-24 21:06** (self-refresh) | `968f977e/571a3fb5` | | `run_68753…` | 07-23 21:44 | **07-24 21:44** (self-refresh) | `64ff94a7/13d3372b` | | `run_11e5…` | 07-24 21:13 | 21:13 (spawn copy) | stale earlier family | Three independently-refreshed families alive at once, each rotation orphaning all other holders. Pre-#202 every session shared one host credentials file, so exactly one family ever existed — that's why this only appeared with the containerization work. Side effect: wipe-on-stop can shred the *only* valid family if the newest rotation happened inside an instance — forced re-login even with nothing else running. `codex` almost certainly carries the same latent bug: same snapshot design (`internal/provider/codex/inject.go`), and OpenAI also rotates refresh tokens. ## Settled design (grilled 2026-07-24) Invariant: **one refresher per provider grant; instances are consumers, never refreshers.** Lab owns the grant without speaking any provider's OAuth protocol. Five decisions from the grilling session: 1. **Refresh clock — blind periodic poke.** A timer periodically runs the provider's refresh-trigger command on the host; the provider's own CLI decides whether the token is near expiry and rotates. Lab detects rotation purely by file change (opaque signature over the credential files), never parsing token contents — no new coupling to any provider's token schema. 2. **Fan-out on rotation — immediate.** When the master's signature changes, re-inject into every live instance HOME through the existing per-provider `InjectCredentials` seam (single call site by design; instance HOMEs are host paths even for container runs). Probe evidence below shows rotation invalidates the old family's access tokens within minutes, so fan-out must directly follow any detected master change. Copies kept fresh never near expiry → instances never self-refresh → no thundering herd. 3. **Adopt-back — eagerly on detection.** If an instance self-refreshes anyway (safety net), the scan detects an instance signature that differs from what lab last injected there and adopts that family into the master **immediately**, then fans out — waiting invites a second instance to fork yet another family. Tie-break when several instances changed in one scan: newest credential-file mtime wins. 4. **Wipe races — adopt-check before every wipe, signature persisted.** The last-injected signature is persisted per run. The stop path synchronously compares signatures → adopts a self-refreshed family → then wipes; startup sweep does the same per orphan home before wiping it. Closes both the stop race and the restart-after-downtime race where an orphan home holds the only valid family. 5. **Seam shape — three new methods on `provider.AgentProvider`:** - `RefreshCredentials(ctx)` — drive the provider's CLI to refresh the master store; - `CredentialsSig(home)` — opaque change-detector over the credential file(s) under a home, `""` meaning the master store (the SpoolSig precedent); - `AdoptCredentials(instanceHome)` — copy an instance's credentials back into the master store. Core owns the rotation loop; adapters own every file path and CLI recipe. Fan-out reuses `InjectCredentials`. Rejected alternatives, for the record: shared credentials file (claude derives the creds path from `CLAUDE_CONFIG_DIR`, which must stay per-run; tmp+rename replaces any symlink/hardlink on first refresh — the agent.sock inode lesson of ADR-0052 — and it reopens the isolation seam #202 closed); sync-back alone (all holders share one expiry clock, so the concurrent-refresh herd at T+8h is the common case); adopt-back-only-on-master-failure (leaves a window up to the remaining master token lifetime where the fleet holds a doomed family). ## Compat probe results (claude, probed live 2026-07-24) - **A real API call is a confirmed non-interactive refresh trigger.** With the master's `expiresAt` forged 60 s into the past, `claude -p --model haiku` (under `CLAUDE_CONFIG_DIR` pointed at the master) succeeded and rewrote the master store in place (23:12:42, 509 → 945 bytes) with a fresh **valid** family — the next spawn ran on it with no re-login. - **The CLI reads credential state from disk before refreshing** — it honored the forged on-disk `expiresAt` rather than any in-memory view. - `claude auth status` does **not** rewrite the credentials file when the token is valid (clean control run). Whether it refreshes an *expired* token was not tested — moot given the confirmed trigger above. - **Rotation invalidates the previous family fast.** A running session still holding the pre-rotation family — its access token hours from expiry — hit "Login expired" within minutes of the master rotation. Consequence for the design: fan-out must directly follow rotation, and an instance mid-request in that window eats one failed request before its re-injected copy takes over. - **Grace-window probe dropped by decision:** deliberately double-spending a refresh token risks the operator's live grant, and eager adopt-back removed the design's dependency on the answer. ## Remaining probes (codex — fold into implementation, not blocking) - What non-interactively triggers a refresh for `codex` (the `codex exec` analogue of `claude -p`)? - Does `codex` read auth state from disk before refreshing? ## References - #202 per-run instance HOME with credential copy/wipe (introduced the snapshot design; its comment already anticipates this fix: "a server-side credential proxy can later replace this copy wholesale") - ADR-0052 container runner (mount inventory; why instance HOMEs are host-writable) - `internal/provider/claudecode/inject.go`, `internal/provider/claudecode/auth.go`, `internal/provider/codex/inject.go`

dominik.polakovics added the

needs-triage

bug

labels

2026-07-24 22:34:43 +02:00

dominik.polakovics changed title from ~~Server-owned credential authority: per-run OAuth snapshots fork the token family, rotation logs everyone out~~ to Server-owned credential authority: per-run OAuth snapshots fork the token family, rotation logs everyone out (grilled 2026-07-24)

2026-07-24 23:45:26 +02:00

dominik.polakovics commented

2026-07-24 23:45:26 +02:00

Author

Owner

This was generated by AI during triage.

Agent Brief

Category: bug
Summary: Make the master credential store the only OAuth rotation site per provider: proactive host-side refresh, immediate fan-out of rotated credentials into live instance HOMEs, eager adopt-back of instance self-refreshes, and adopt-checks before every credential wipe.

Current behavior:
Each run gets a one-time snapshot of the master credential store into its private instance HOME at spawn, with no write-back and no later updates. When a long-lived instance's access token expires, its agent CLI refreshes its private copy, minting a new token family that invalidates the refresh token held by the master and every other snapshot. The host then reads logged-out, the spawn gate refuses new runs, and wipe-on-stop can destroy the only still-valid family. Both the claude-code and codex adapters share this snapshot design.

Desired behavior:
Invariant: one refresher per provider grant; instances are consumers, never refreshers.

A lab-owned loop keeps the master store fresh with a blind periodic poke: it periodically drives the provider's own CLI (for claude-code, a real API call — claude -p — is the probe-confirmed non-interactive trigger; the CLI decides whether to rotate). Lab never parses token contents and never speaks any provider's OAuth protocol.
Lab detects master rotation via an opaque per-provider credential signature. On any master change, it immediately re-injects credentials into every live instance HOME (rotation invalidates the old family's access tokens within minutes — probed live — so fan-out must directly follow detection). Instance HOMEs are host paths even for container runs, so host-side writes suffice.
If a scan finds an instance whose credential signature differs from what lab last injected there (a self-refresh), that family is adopted into the master immediately and fanned out. When several instances changed in one scan, the newest credential-file mtime wins.
The last-injected signature is persisted per run. Before any credential wipe — the stop path and the startup sweep of orphan homes — lab compares signatures and adopts a self-refreshed family into the master first, so a wipe can never destroy the only valid family.
Instances whose copies are kept fresh never near expiry, so they never self-refresh — adopt-back is a safety net, not the mechanism.

Key interfaces:

provider.AgentProvider grows three methods (settled in grilling):
- RefreshCredentials(ctx context.Context) error — drive the provider's CLI to refresh the master store, non-interactively, on the host;
- CredentialsSig(home string) string — opaque change-detector over the credential file(s) under a home, "" meaning the master store (follow the existing SpoolSig precedent for shape and semantics);
- AdoptCredentials(instanceHome string) error — copy an instance HOME's credentials back into the master store.
Core owns the rotation loop (schedule, sig comparison, fan-out ordering, adopt tie-breaks); adapters own every file path and CLI recipe. Fan-out reuses the existing InjectCredentials seam — it must remain the single injection call site.
The per-run last-injected signature is persisted (run row or a sidecar in the run's state dir) so the startup sweep can adopt-check orphan homes after lab downtime.
Both registered providers (claude-code and codex) implement all three methods; the providertest/conformance surface pins the contract. The codex refresh-trigger recipe needs a live probe during implementation (candidate: a minimal codex exec call, the analogue of claude -p); record the confirmed recipe in the codex compat notes.
The auth.go stance stands: no lab code parses token schemas or reimplements OAuth; everything credential-shaped stays adapter-private and CLI-mediated.

Acceptance criteria:

With multiple live instance HOMEs, a master rotation is followed by every live home holding the new credentials via re-injection, without any instance self-refreshing (fake-backed test at the seam).
The rotation loop periodically invokes RefreshCredentials and triggers fan-out exactly when the master signature changes — no fan-out on a no-op poke.
A simulated instance self-refresh is detected by signature mismatch against the persisted last-injected value, adopted into the master, and fanned out to other live homes; with several changed homes, the newest mtime wins.
Stopping a run whose home holds a self-refreshed family adopts that family into the master before the wipe; the startup sweep does the same for orphan homes.
Lab core never reads credential file contents — signatures are opaque strings produced by the adapter.
Both claudecode and codex implement the three new methods with test coverage; the conformance suite pins the ""-means-master convention for CredentialsSig.
The codex non-interactive refresh trigger is probed live and the working recipe recorded in the codex compat notes (blocking only that adapter's RefreshCredentials, not the core loop).

Out of scope:

Containerized provider login flow (#206) — login stays a host-level operator action.
A server-side credential proxy replacing the copy design (anticipated as a later evolution in #202's notes) — the snapshot-plus-fan-out model stays.
The server-side refresh-token grace-window question — deliberately dropped: probing it double-spends a live refresh token, and eager adopt-back removed the design's dependency on the answer.
Provider auth flows that are not file-based OAuth (api-key, external) — no behavior change for them.

> *This was generated by AI during triage.* ## Agent Brief **Category:** bug **Summary:** Make the master credential store the only OAuth rotation site per provider: proactive host-side refresh, immediate fan-out of rotated credentials into live instance HOMEs, eager adopt-back of instance self-refreshes, and adopt-checks before every credential wipe. **Current behavior:** Each run gets a one-time snapshot of the master credential store into its private instance HOME at spawn, with no write-back and no later updates. When a long-lived instance's access token expires, its agent CLI refreshes its private copy, minting a new token family that invalidates the refresh token held by the master and every other snapshot. The host then reads logged-out, the spawn gate refuses new runs, and wipe-on-stop can destroy the only still-valid family. Both the claude-code and codex adapters share this snapshot design. **Desired behavior:** Invariant: one refresher per provider grant; instances are consumers, never refreshers. 1. A lab-owned loop keeps the master store fresh with a blind periodic poke: it periodically drives the provider's own CLI (for claude-code, a real API call — `claude -p` — is the probe-confirmed non-interactive trigger; the CLI decides whether to rotate). Lab never parses token contents and never speaks any provider's OAuth protocol. 2. Lab detects master rotation via an opaque per-provider credential signature. On any master change, it immediately re-injects credentials into every live instance HOME (rotation invalidates the old family's access tokens within minutes — probed live — so fan-out must directly follow detection). Instance HOMEs are host paths even for container runs, so host-side writes suffice. 3. If a scan finds an instance whose credential signature differs from what lab last injected there (a self-refresh), that family is adopted into the master immediately and fanned out. When several instances changed in one scan, the newest credential-file mtime wins. 4. The last-injected signature is persisted per run. Before any credential wipe — the stop path and the startup sweep of orphan homes — lab compares signatures and adopts a self-refreshed family into the master first, so a wipe can never destroy the only valid family. 5. Instances whose copies are kept fresh never near expiry, so they never self-refresh — adopt-back is a safety net, not the mechanism. **Key interfaces:** - `provider.AgentProvider` grows three methods (settled in grilling): - `RefreshCredentials(ctx context.Context) error` — drive the provider's CLI to refresh the master store, non-interactively, on the host; - `CredentialsSig(home string) string` — opaque change-detector over the credential file(s) under a home, `""` meaning the master store (follow the existing SpoolSig precedent for shape and semantics); - `AdoptCredentials(instanceHome string) error` — copy an instance HOME's credentials back into the master store. - Core owns the rotation loop (schedule, sig comparison, fan-out ordering, adopt tie-breaks); adapters own every file path and CLI recipe. Fan-out reuses the existing `InjectCredentials` seam — it must remain the single injection call site. - The per-run last-injected signature is persisted (run row or a sidecar in the run's state dir) so the startup sweep can adopt-check orphan homes after lab downtime. - Both registered providers (claude-code and codex) implement all three methods; the providertest/conformance surface pins the contract. The codex refresh-trigger recipe needs a live probe during implementation (candidate: a minimal `codex exec` call, the analogue of `claude -p`); record the confirmed recipe in the codex compat notes. - The auth.go stance stands: no lab code parses token schemas or reimplements OAuth; everything credential-shaped stays adapter-private and CLI-mediated. **Acceptance criteria:** - [ ] With multiple live instance HOMEs, a master rotation is followed by every live home holding the new credentials via re-injection, without any instance self-refreshing (fake-backed test at the seam). - [ ] The rotation loop periodically invokes `RefreshCredentials` and triggers fan-out exactly when the master signature changes — no fan-out on a no-op poke. - [ ] A simulated instance self-refresh is detected by signature mismatch against the persisted last-injected value, adopted into the master, and fanned out to other live homes; with several changed homes, the newest mtime wins. - [ ] Stopping a run whose home holds a self-refreshed family adopts that family into the master before the wipe; the startup sweep does the same for orphan homes. - [ ] Lab core never reads credential file contents — signatures are opaque strings produced by the adapter. - [ ] Both claudecode and codex implement the three new methods with test coverage; the conformance suite pins the `""`-means-master convention for `CredentialsSig`. - [ ] The codex non-interactive refresh trigger is probed live and the working recipe recorded in the codex compat notes (blocking only that adapter's `RefreshCredentials`, not the core loop). **Out of scope:** - Containerized provider login flow (#206) — login stays a host-level operator action. - A server-side credential proxy replacing the copy design (anticipated as a later evolution in #202's notes) — the snapshot-plus-fan-out model stays. - The server-side refresh-token grace-window question — deliberately dropped: probing it double-spends a live refresh token, and eager adopt-back removed the design's dependency on the answer. - Provider auth flows that are not file-based OAuth (api-key, external) — no behavior change for them.

dominik.polakovics added

ready-for-agent

and removed

needs-triage

labels

2026-07-24 23:45:32 +02:00

dominik.polakovics referenced this issue from a commit

2026-07-25 00:37:10 +02:00

fix(provider,core): server-owned credential authority — one refresher per grant (ADR-0055)

dominik.polakovics referenced this issue from a pull request that will close it,

2026-07-25 00:37:36 +02:00

Server-owned credential authority: one refresher per provider grant (ADR-0055) #223