fix(podmanx,nix): move payload cgroup delegation to a never-restarted lab-payload.service holder (ADR-0059) #238

Merged

dominik.polakovics merged 1 commit from afk/237 into main

2026-07-26 01:22:56 +02:00

dominik.polakovics commented

2026-07-26 00:57:52 +02:00

Owner

What

Every lab.service restart after the ADR-0058 deploy EBUSY-looped on dev-new because DelegateSubgroup= does not protect the main-process spawn: systemd clone3s the new main PID into the unit cgroup root (exec_params_needs_control_subcgroup requires EXEC_IS_CONTROL, which the ExecStart process never carries) and the EBUSY is not retried — so with KillMode=process survivors keeping the delegating root populated, the trap re-armed on every restart. Any layout keeping controllers on lab.service's own cgroup root across restarts is structurally dead.

Per the maintainer-approved direction, the delegated payload cgroup moves out of lab.service into a static, never-restarted holder unit:

lab-payload.service (new, container.enable-gated): Type=exec + sleep infinity, lab service user, Delegate=yes, DelegateSubgroup=main, KillMode=process, restartIfChanged = false. Containers land under /system.slice/lab-payload.service/payload via --cgroup-parent in all three spawn shapes (sessions, login panes, CLI pokes) — Result.CgroupParent() contract unchanged, value changed.
lab.service loses Delegate/DelegateSubgroup entirely; gains Wants=/After= on the holder (Wants, not Requires: a missing holder degrades to an actionable preflight refusal).
Preflight: new cgroup-holder check (retires cgroup-layout) verifies the holder cgroup is delegated/writable, with hints naming lab-payload.service; setupCgroups adopts holder-root strays into main/, enables +memory +pids on the holder root and confirms the controllers arrive in payload/ (cgroup-delegation, name kept).
Per-spawn restart-safety guard (cgroup-restart-safety, name kept): asserts lab.service's own subtree_control stays empty (the re-armed trap — issue acceptance criterion) and payload/ still carries memory+pids (a restarted holder would otherwise silently void the caps — the #205 no-false-green lesson).
lab-cgroup-hygiene retargeted: clears all delegation under /sys/fs/cgroup/system.slice/lab.service including the root's own subtree_control — this is also the migration path off the ADR-0058 layout on the first post-upgrade start (accepted cost: pre-0059 survivor containers lose their caps, as in ADR-0058's migration).
ADR-0059 records the design with the systemd source receipt; ADR-0058 marked superseded in part; docs/ops.md updated.
flake module-eval check: negative guards that lab.service never delegates (both variants), holder-unit assertions (Delegate, DelegateSubgroup=main, KillMode=process, User=lab, sleep ExecStart), eval-level restartIfChanged == false, holder absent under container.enable = false, hygiene retarget pinned.

Verification

go build ./..., go test ./..., gofmt -l all green (run locally with go1.26.0; toolchain absent from the sandbox by default). Unit tests cover the new layout against fake Deps: green path, holder missing → cgroup-holder naming the unit, re-armed trap on lab's own root, stray adoption into holder main/, holder-lost-delegation at Verify time, vacuous verify.
golangci-lint and nix flake check are not available in this sandbox — CI must be the gate for lint and the nixos-module eval check.
Full end-to-end restart validation (live containers + tmux survivors across repeated systemctl restart lab) needs a real NixOS host — dev-new deploy by the maintainer, per the issue's validation note.

Closes #237

🤖 Generated with Claude Code

## What Every `lab.service` restart after the ADR-0058 deploy EBUSY-looped on dev-new because `DelegateSubgroup=` does **not** protect the main-process spawn: systemd clone3s the new main PID into the unit cgroup root (`exec_params_needs_control_subcgroup` requires `EXEC_IS_CONTROL`, which the `ExecStart` process never carries) and the EBUSY is not retried — so with `KillMode=process` survivors keeping the delegating root populated, the trap re-armed on every restart. Any layout keeping controllers on `lab.service`'s own cgroup root across restarts is structurally dead. Per the maintainer-approved direction, the delegated payload cgroup moves out of `lab.service` into a static, **never-restarted** holder unit: - **`lab-payload.service`** (new, `container.enable`-gated): `Type=exec` + `sleep infinity`, lab service user, `Delegate=yes`, `DelegateSubgroup=main`, `KillMode=process`, `restartIfChanged = false`. Containers land under `/system.slice/lab-payload.service/payload` via `--cgroup-parent` in all three spawn shapes (sessions, login panes, CLI pokes) — `Result.CgroupParent()` contract unchanged, value changed. - **`lab.service`** loses `Delegate`/`DelegateSubgroup` entirely; gains `Wants=`/`After=` on the holder (Wants, not Requires: a missing holder degrades to an actionable preflight refusal). - **Preflight**: new `cgroup-holder` check (retires `cgroup-layout`) verifies the holder cgroup is delegated/writable, with hints naming `lab-payload.service`; `setupCgroups` adopts holder-root strays into `main/`, enables `+memory +pids` on the holder root and confirms the controllers arrive in `payload/` (`cgroup-delegation`, name kept). - **Per-spawn restart-safety guard** (`cgroup-restart-safety`, name kept): asserts `lab.service`'s own `subtree_control` stays empty (the re-armed trap — issue acceptance criterion) and `payload/` still carries memory+pids (a restarted holder would otherwise silently void the caps — the #205 no-false-green lesson). - **`lab-cgroup-hygiene`** retargeted: clears **all** delegation under `/sys/fs/cgroup/system.slice/lab.service` including the root's own `subtree_control` — this is also the migration path off the ADR-0058 layout on the first post-upgrade start (accepted cost: pre-0059 survivor containers lose their caps, as in ADR-0058's migration). - **ADR-0059** records the design with the systemd source receipt; ADR-0058 marked superseded in part; `docs/ops.md` updated. - **flake module-eval check**: negative guards that `lab.service` never delegates (both variants), holder-unit assertions (`Delegate`, `DelegateSubgroup=main`, `KillMode=process`, `User=lab`, sleep ExecStart), eval-level `restartIfChanged == false`, holder absent under `container.enable = false`, hygiene retarget pinned. ## Verification - `go build ./...`, `go test ./...`, `gofmt -l` all green (run locally with go1.26.0; toolchain absent from the sandbox by default). Unit tests cover the new layout against fake `Deps`: green path, holder missing → `cgroup-holder` naming the unit, re-armed trap on lab's own root, stray adoption into holder `main/`, holder-lost-delegation at Verify time, vacuous verify. - `golangci-lint` and `nix flake check` are **not available in this sandbox** — CI must be the gate for lint and the nixos-module eval check. - Full end-to-end restart validation (live containers + tmux survivors across repeated `systemctl restart lab`) needs a real NixOS host — dev-new deploy by the maintainer, per the issue's validation note. Closes #237 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dominik.polakovics added 1 commit

2026-07-26 00:57:52 +02:00

fix(podmanx,nix): move payload cgroup delegation to a never-restarted holder unit (ADR-0059)

ci / native (pull_request) Successful in 10m7s

Details

ci-nix / flake-check (pull_request) Successful in 20m42s

Details

be02171559

DelegateSubgroup= does not protect the main-process spawn: systemd
clone3(CLONE_INTO_CGROUP)s the new main PID into the unit cgroup root
(exec_params_needs_control_subcgroup requires EXEC_IS_CONTROL, which the
ExecStart process never carries) and the EBUSY is not retried — so with
KillMode=process survivors keeping lab.service's delegating root
populated, every restart after any container ran failed with "Failed to
spawn executor: Device or resource busy" (the 2026-07-26 dev-new
incident). Any layout that keeps controllers on lab.service's own root
across restarts is structurally dead; ADR-0058's placement is retired.

The delegation moves into a static lab-payload.service holder unit
(Delegate=yes, DelegateSubgroup=main, KillMode=process, User=lab,
sleep infinity, NixOS restartIfChanged=false — it never restarts, so
the spawn trap can never fire for it). lab.service drops Delegate/
DelegateSubgroup entirely and gains Wants=/After= on the holder;
containers now land under /system.slice/lab-payload.service/payload via
--cgroup-parent in all three spawn shapes.

Preflight retargets: the new cgroup-holder check (replacing the retired
cgroup-layout) verifies the holder is delegated to the service user,
setupCgroups adopts holder-root strays into main/, enables
+memory +pids on the holder root and confirms the controllers arrive in
payload/ (cgroup-delegation, name kept). The per-spawn restart-safety
guard now asserts lab.service's own subtree_control stays empty (the
re-armed trap) and payload/ still carries memory+pids (a restarted
holder would silently void the caps). The lab-cgroup-hygiene oneshot is
retargeted to clear ALL delegation under lab.service's cgroup including
the root's own — both the migration off the ADR-0058 layout and
standing insurance.

ADR-0059 records the design with the systemd source receipt; ADR-0058
is marked superseded in part; ops.md updated. The flake's module-eval
check now asserts lab.service never delegates, the holder's invariants,
and its absence under container.enable = false.

Closes #237

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics commented

2026-07-26 01:19:14 +02:00

Author

Owner

[autoland] verdict: pass

dominik.polakovics merged commit c2865f4eee into main

2026-07-26 01:22:56 +02:00

dominik.polakovics deleted branch afk/237

2026-07-26 01:22:56 +02:00

dominik.polakovics referenced this pull request from a commit

2026-07-26 01:22:58 +02:00

Merge pull request 'fix(podmanx,nix): move payload cgroup delegation to a never-restarted lab-payload.service holder (ADR-0059)' (#238) from afk/237 into main