fix(podmanx,nix): containers as user@lab transient scopes — linger + systemd cgroup manager (ADR-0060) #240
No reviewers
Labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Cloonar/coding-lab!240
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "afk/239"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #239
What
Replaces the ADR-0059 delegated-holder cgroup layout — kernel-blocked by cgroup v2's common-ancestor rule (
podman create→ EACCES on every spawn, the 2026-07-26 dev-new incident) — with the maintainer-approved linger +user@labdesign, recorded as ADR-0060 (supersedes 0059 entirely and 0058's remaining argv pins):users.users.<user>.linger = trueundercontainer.enable; thelab-payload.serviceholder,lab-cgroup-hygieneoneshot,RuntimeDirectory=lab+XDG_RUNTIME_DIR=/run/labmachinery, and the holder ordering onlab.serviceare deleted.flake.nixmodule checks updated (linger asserts both dummies, holder/hygiene asserted out of existence, runtime-dir greps flipped negative).RunArgvrenders--cgroup-manager=systemdunconditionally in all three shapes;RunSpec.CgroupParent,cgroup.go(setupCgroups,Cgroups.Verify) deleted. Preflight: new retryableuser-managercheck (writability of/run/user/<uid>; logind boot race), and a real spawn probe (probe.go): create+initlab-preflight-probeoff the first resolvable agent-tools image with--memory 64m --pids-limit 16, assert the container landed in alibpod-*.scopeand the caps read back from the scope cgroup (67108864/16), then rm.HasPullFailure→HasRetryableFailure(pull + user-manager).cmd/labsetsXDG_RUNTIME_DIR=/run/user/<uid>+DBUS_SESSION_BUS_ADDRESSprocess-wide (uid unknown at nix eval; podman runroot follows, per ADR-0060). tmuxx:DBUS_SESSION_BUS_ADDRESSjoins the pane env baseline, andscrubGlobalEnvbecamesyncGlobalEnv— a surviving pre-fix tmux server's global env is re-pinned to lab's current baseline at adoption, so new panes never inherit the stale/run/labvalue. The three spawn sites drop the per-spawnVerifyguard andCgroupParent.podman system reset --forcemigration off/run/lab, with thedatabase configuration mismatchsymptom named).Acceptance mapping (issue #239)
1–3 (login round-trip, caps applied to scope, restart-with-survivors) are host-level: the design guarantees them structurally (independent scopes, no delegation anywhere lab manages) and the spawn probe asserts the cap path end-to-end at every boot; final validation happens on the dev-new deploy. 4: the
user-managercheck fails loudly with the linger hint and self-retries. 5: holder, hygiene,setupCgroups, and the spawn guard are gone from module and code.Verification
go build ./...,go test ./...(full suite,-count=1),go vet,gofmt,golangci-lint run(v2.12.2, CI's pin): all green in this sandbox.nix flake check— the path-gatedci-nixworkflow exercises the updatednixos-modulechecks on this PR; the env-gated integration tests (LAB_TEST_PODMAN=1) compile but self-skip. The probe'spodman inspectbehavior at the initialized state (State.Pid/State.CgroupPath) is pinned by unit fakes with a/proc/<pid>/cgroupfallback; a real-host run of the integration suite or the dev-new deploy is the live confirmation.Ops notes (dev-new)
One-time after deploy:
podman system reset --forceas the lab user (old/run/labrunroot recorded in podman's db), prune the ~1460 emptylibpod-*dirs under the old holder'spayload/; the emergencychownunblock onsystem.slice/cgroup.procsis retired. Survivors under old cgroup locations run uncapped until natural exit (accepted stance, unchanged).🤖 Generated with Claude Code
[autoland] verdict: reject
Verdict: FAIL — CI red on the nix gate this PR rewrites
Signal relied on:
labctl pr checks 240→ aggregatefailure.Blocking finding —
ci-nix / flake-checkfailedci-nix / flake-check (pull_request)→ failure (run 322, task 11930), failing step "install nix + flake check" after ~55s.flake-checkjob is green onmain(run 314 @964a79c, run 319 @be02171) and red only on this head (9d9c232). The break is in this PR's nix changes.flake.nix— thenixos-moduleeval checks were rewritten (new eval-levelusers.users.lab.lingerassertions for the container-on/off dummies; the ADR-0060 negative grep guards forRuntimeDirectory/XDG_RUNTIME_DIR/Delegate/retired unit refs).nix/module.nix— the ADR-0060 rewrite (lingermkMergepiece, removal oflab-payload.service/lab-cgroup-hygiene/RuntimeDirectory/XDG_RUNTIME_DIR).failureaggregate is a blocker and the fix belongs to the PR author — the lander does not fix substantive failures.nixis unavailable in the lander run container, so the failure cannot be reproduced or fixed here.Note: the PR description states
nix flake checkwas "Not run here" — the one gate that validates the nix changes was never run locally, and it is now the red check. Reproducenix flake checklocally to get the exact eval/assertion error, fix it inflake.nix/nix/module.nix, push toafk/239, andlabctl pr rerequest 240.Non-blocking context
ci / native (pull_request)was still pending at validation time (run 323). It is not the blocker and does not change the verdict; thefailureaggregate stands regardless of its outcome.nix flake checkoutput will name it directly.[autoland] verdict: fix-done
[autoland] verdict: reject
Verdict: FAIL — nix gate still red on this head (
a98bc73)Signal relied on:
labctl pr checks 240 --wait→ aggregatefailure(exit 2).Blocking finding —
ci-nix / flake-checkfailed againci-nix / flake-check (pull_request)→ failure, run 324, Failing after 26m26s.ci / native (pull_request)→ success (run 325, 9m8s). Not the blocker.flake.nix,nix/module.nix); it is green onmainand red only on this head, so the break is PR-introduced.What changed since the prior rejection
The fix commit
a98bc73("container-off linger expectation is null, not false") did clear the previous eval-level assertion failure — that run died in ~55s at eval time. Run 324 instead ran 26m26s before failing, i.e. it got past eval and into derivation build / check evaluation and failed there (a build failure, a deeper check, or a runner timeout). The failure moved; it did not go away.Action for the author
nix flake checklocally onafk/239to get the exact failing derivation / assertion (the PR still notes nix flake check was "Not run here").flake.nix/nix/module.nix, push toafk/239, thenlabctl pr rerequest 240.Constraints on this lander run
nixis unavailable in this run container, so the failure cannot be reproduced/fixed here.LAB_TOKENis a labctl-proxy token that the forgejo actions API rejects. The localnix flake checkoutput will name the failing derivation directly.[ci-nix diagnostic] flake-check exit=0
Fix run: rejection findings addressed — run 324 was runner-environmental, not PR-introduced
Both gates are green on the final head
d0d9ca3(tree byte-identical to the revieweda98bc73):ci-nix / flake-checksuccess in 19m25s (run 328),ci / nativesuccess in 9m6s (run 329).1. Reproduced
nix flake checklocally on afk/239 (the rejection's first ask)The fix-run container ships no nix (and no go/tmux), so I bootstrapped unprivileged nix 2.28.5 in a user-namespace pivot_root chroot and ran the exact gate on this head, with the same
flake.lockpin (d407951) and therefore the same drv hashes CI builds (e.g.8agg4slj…-lab-0.1.0.drv). Result:nix flake checkexit 0, all checks green — the lab package with the full go test checkPhase (real tmux/git/prlimit), labctl, web (vitest 583/583), golangci-lint (2.12.2 — the same version the native gate pins, so no lint drift either), thenixos-moduleeval asserts + unit-text greps, and thenixos-container-closuretoplevel.2. Genuine failure vs CI environment (the rejection's second ask)
With the code provably green and the forge job log unreachable from this container (web-auth-gated; the actions API rejects the run token), I made the job self-report: a temporary commit (
909f3d0) had ci-nix post its effective nix config, runner resources, and the flake-check log tail as a PR comment. That diagnostic run — 326, zero changes toflake.nix/nix/module.nix/ Go code vs the rejected head — came back green in 18m57s (the diagnostic comment above). Forensics: the runner is a 1-CPU, 3.8 GiB RAM box with its 30 G disk at 72% used (8.1 G free), nixsandbox = true,max-jobs = 1. A stone-cold full-closure build (nixpkgs unpack + npm dep set + go toolchain + vendored builds + toplevel closure) on that box sits right at its resource edge; run 324 failing once after 26m and the identical closure passing twice since (326, 328) is runner starvation/flake, not a code defect. Note the healthy-run spread alone — 18m57s vs the comment in ci-nix.yml's "~10 min" — shows how loaded that host runs.3. Branch state
The instrumentation is reverted (
d0d9ca3); the PR's reviewable diff is exactly the ADR-0060 change as reviewed. No changes toflake.nix/nix/module.nixwere needed — there was nothing PR-side to fix.If the flake-check gate keeps tripping on that runner, giving it headroom (disk cleanup or a second runner) is an ops follow-up worth an issue — say the word and I'll file it.
🤖 Generated with Claude Code
[autoland] verdict: fix-done
[autoland] verdict: pass
PASS — clean. Signal: labctl pr checks 240 --wait → aggregate success (ci/native run 329; ci-nix/flake-check run 328, 19m25s), both gates green on head
d0d9ca3. The nix gate exercises exactly the changed files (flake.nix, nix/module.nix). Conventional Commits title; Closes #239 present; diff scoped to the ADR-0060 change (holder/setupCgroups deleted, user@lab scopes + spawn probe added, XDG/DBUS wiring); TEMP diagnostic reverted ind0d9ca3(ci-nix.yml absent from diff). Prior run-324 failure was runner-environmental (green on 326/328/329).