fix(nix,podmanx): containers as user@lab scopes — linger + systemd cgroup manager; cross-unit payload attach is kernel-blocked (EACCES), holder still carries the restart trap #239

New issue

Closed

opened 2026-07-26 10:14:08 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-07-26 10:14:08 +02:00

Owner

| This was generated by AI during a live diagnosis session with the maintainer.

Incident

2026-07-26, dev-new: after the issue #237 holder-unit fix (deployed ~01:32 as "ADR-0059") every container spawn fails at podman create — provider login panes return 500 (start login session: session "lab-login-claude-code" exited immediately), and agent-run containers are equally dead. Journal signature, one per attempt:

conmon <id> <nwarn>: runtime stderr: write to `/sys/fs/cgroup/system.slice/lab-payload.service/payload/libpod-<id>/cgroup.procs`: Permission denied
conmon <id> <error>: Failed to create container: exit status 1

~1460 leftover empty libpod-* dirs have accumulated under payload/ from the failed creates (harmless, prune during cleanup).

Diagnosis (verified live on dev-new)

The deployed layout is exactly as #237 specified and is not misconfigured: lab-payload.service has Delegate=yes, User=lab, its cgroup root + cgroup.procs + cgroup.subtree_control are lab-owned, payload/ is lab-owned with cpuset cpu io memory pids enabled, and lab.service has Delegate=no. The destination files crun writes are writable by lab.

The failure is the cgroup-v2 delegation containment rule: to migrate a PID between cgroups, a non-root writer needs write access to cgroup.procs of the common ancestor of source and destination — not just the destination. crun runs inside lab.service; the target is lab-payload.service; the common ancestor is system.slice, whose cgroup.procs is root-owned → EACCES on every attach.

Probes (all as described, reproduced deterministically):

lab-user write of a session PID into a fresh lab-owned cgroup under payload/ → Permission denied.
Identical write as root → succeeds.
lab-user move of a PID whose source is already inside the lab-payload.service subtree → succeeds.

Probe 3 is the smoking gun: destination permissions are fine; only the cross-delegation ancestor check fails.

Why the holder cannot be patched (do not re-attempt these)

The restart-EBUSY trap from #237 still exists — it just moved into the holder. lab-payload.service runs KillMode=process, Restart=no, X-RestartIfChanged=false, DelegateSubgroup=main, with controllers enabled on its cgroup root. Any restart while containers survive (manual restart, the sleep dying, any future unit edit) hits the same executor-spawn-into-root EBUSY, because DelegateSubgroup= does not apply to the main-process spawn (verified in systemd v260 exec_params_needs_control_subcgroup(), see #237). "Never restart it" is a pin, not a fix.
podman system service (REST API) does not help. The invariant is: an unprivileged process can only attach PIDs into a delegated subtree it already lives in. So the API service would have to live inside the payload unit — making it just another spawner carrying the same restart trap, and a worse one: it idle-exits, is socket-activated, and its binary changes every deploy.
A custom spawner daemon in the holder — same analysis, same trap, plus an un-updatable binary.

General form: whoever spawns containers must share the delegated subtree, and whoever shares the subtree cannot safely restart with survivors. Only root/systemd escapes this. Both architectures tried so far (ADR-0058 in-unit payload → restart-EBUSY; #237/ADR-0059 cross-unit payload → spawn-EACCES) are two sides of the same kernel wall.

Decided fix (maintainer-approved 2026-07-26 — don't re-grill the direction)

Linger + user@lab manager: stop performing cgroup placement as an unprivileged process; let systemd do it. This is ADR-0058's documented fallback, now promoted to the primary design:

users.users.lab.linger = true → permanent user@lab.service user manager for the lab user. lab.service itself stays a system service running as lab.
Podman stays rootless but switches from --cgroup-parent=…/payload --cgroup-manager=cgroupfs to --cgroup-manager=systemd in all three shapes (run pane, login pane, non-interactive CLI poke — one renderer, podmanx.RunArgv). Each container becomes its own transient scope (libpod-<id>.scope) under user@lab.service, created via the lab user manager; cross-delegation attaches are forwarded by the user manager to PID 1 (AttachProcessesToUnit), so the ancestor rule never applies. --memory/--pids-limit land as per-scope caps.
Wiring: podman invocations need the lab user manager reachable — XDG_RUNTIME_DIR=/run/user/<lab-uid> and DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/<lab-uid>/bus. This re-opens ADR-0057's XDG_RUNTIME_DIR=/run/lab + RuntimeDirectoryPreserve decisions: decide whether podman's runroot/tmpdir move to /run/user/<uid> (linger keeps it alive, so RuntimeDirectoryPreserve machinery may become unnecessary) and document in the ADR.
Delete the whole hand-rolled layout: lab-payload.service holder, lab-cgroup-hygiene oneshot, podmanx.setupCgroups, the per-spawn restart-safety guard, and Delegate/DelegateSubgroup on lab.service. lab.service restarts can then never EBUSY, and containers survive lab restarts by construction (independent scope units).
Preflight becomes a real spawn probe. The current preflight was false-green: it verifies controllers/ownership but never performs an actual PID migration — the one operation that fails. Replace with: spawn a trivial container with --memory set, assert it starts and the cap landed in its scope cgroup. That catches this whole class regardless of layout (preserves #205's no-false-green property for real).

Structural wins: each failure mode disappears rather than being pinned — no unit we manage carries delegation, user@lab is systemd's own machinery (never restarted by nixos-rebuild, and stopping it kills its whole subtree, so it has no survivor-EBUSY mode either).

Migration / repo state notes

ADR-0059 and the holder implementation are deployed on dev-new but are NOT on repo main (repo main still has the ADR-0058 in-unit layout; docs/adr/ ends at 0058). Locate that branch or supersede in place; the new ADR should take the next free number and note the phantom deployed "ADR-0059".
Survivor containers under the old cgroup locations keep running uncapped until natural exit — accepted, same stance as ADR-0058's migration note. Prune the ~1460 empty libpod-* leftovers under the holder's payload/, then the holder unit can be removed (its cgroup dies with its last survivor).
Emergency unblock currently available on dev-new until this lands (documented, applied at the maintainer's discretion): chown lab /sys/fs/cgroup/system.slice/cgroup.procs satisfies the ancestor check; holds until systemd re-realizes the slice.

Acceptance

Provider login round-trip works (login pane container starts, /auth/login/start 200).
Agent run containers start with --memory/--pids-limit verifiably applied to their scope cgroup.
systemctl restart lab with live containers: lab comes back clean, containers survive, and a new container can be spawned afterwards.
Preflight fails loudly (actionable message) when the user manager is unreachable, instead of false-green.
lab-payload.service, lab-cgroup-hygiene, setupCgroups, and the spawn guard are gone from module + code.

References: #237 (root-cause of the restart side, closed), ADR-0052, ADR-0057, ADR-0058 (fallback promoted by this issue), deployed-only "ADR-0059".

| *This was generated by AI during a live diagnosis session with the maintainer.* ## Incident 2026-07-26, dev-new: after the issue #237 holder-unit fix (deployed ~01:32 as "ADR-0059") **every container spawn fails at `podman create`** — provider login panes return 500 (`start login session: session "lab-login-claude-code" exited immediately`), and agent-run containers are equally dead. Journal signature, one per attempt: ``` conmon <id> <nwarn>: runtime stderr: write to `/sys/fs/cgroup/system.slice/lab-payload.service/payload/libpod-<id>/cgroup.procs`: Permission denied conmon <id> <error>: Failed to create container: exit status 1 ``` ~1460 leftover empty `libpod-*` dirs have accumulated under `payload/` from the failed creates (harmless, prune during cleanup). ## Diagnosis (verified live on dev-new) The deployed layout is **exactly as #237 specified** and is not misconfigured: `lab-payload.service` has `Delegate=yes`, `User=lab`, its cgroup root + `cgroup.procs` + `cgroup.subtree_control` are lab-owned, `payload/` is lab-owned with `cpuset cpu io memory pids` enabled, and `lab.service` has `Delegate=no`. The destination files crun writes are writable by lab. The failure is the **cgroup-v2 delegation containment rule**: to migrate a PID between cgroups, a non-root writer needs write access to `cgroup.procs` of the *common ancestor* of source and destination — not just the destination. crun runs inside `lab.service`; the target is `lab-payload.service`; the common ancestor is `system.slice`, whose `cgroup.procs` is root-owned → EACCES on every attach. Probes (all as described, reproduced deterministically): 1. lab-user write of a session PID into a fresh **lab-owned** cgroup under `payload/` → `Permission denied`. 2. Identical write as root → succeeds. 3. lab-user move of a PID whose source is **already inside** the `lab-payload.service` subtree → succeeds. Probe 3 is the smoking gun: destination permissions are fine; only the cross-delegation ancestor check fails. ## Why the holder cannot be patched (do not re-attempt these) - **The restart-EBUSY trap from #237 still exists — it just moved into the holder.** `lab-payload.service` runs `KillMode=process`, `Restart=no`, `X-RestartIfChanged=false`, `DelegateSubgroup=main`, with controllers enabled on its cgroup root. Any restart while containers survive (manual restart, the `sleep` dying, any future unit edit) hits the same executor-spawn-into-root EBUSY, because `DelegateSubgroup=` does not apply to the main-process spawn (verified in systemd v260 `exec_params_needs_control_subcgroup()`, see #237). "Never restart it" is a pin, not a fix. - **`podman system service` (REST API) does not help.** The invariant is: an unprivileged process can only attach PIDs into a delegated subtree it already lives in. So the API service would have to live inside the payload unit — making it just another spawner carrying the same restart trap, and a worse one: it idle-exits, is socket-activated, and its binary changes every deploy. - **A custom spawner daemon in the holder** — same analysis, same trap, plus an un-updatable binary. General form: whoever spawns containers must share the delegated subtree, and whoever shares the subtree cannot safely restart with survivors. Only root/systemd escapes this. Both architectures tried so far (ADR-0058 in-unit payload → restart-EBUSY; #237/ADR-0059 cross-unit payload → spawn-EACCES) are two sides of the same kernel wall. ## Decided fix (maintainer-approved 2026-07-26 — don't re-grill the direction) **Linger + `user@lab` manager: stop performing cgroup placement as an unprivileged process; let systemd do it.** This is ADR-0058's documented fallback, now promoted to the primary design: - `users.users.lab.linger = true` → permanent `user@lab.service` user manager for the lab user. `lab.service` itself stays a system service running as lab. - Podman stays rootless but switches from `--cgroup-parent=…/payload --cgroup-manager=cgroupfs` to `--cgroup-manager=systemd` in **all three shapes** (run pane, login pane, non-interactive CLI poke — one renderer, `podmanx.RunArgv`). Each container becomes its own transient scope (`libpod-<id>.scope`) under `user@lab.service`, created via the lab user manager; cross-delegation attaches are forwarded by the user manager to PID 1 (`AttachProcessesToUnit`), so the ancestor rule never applies. `--memory`/`--pids-limit` land as per-scope caps. - Wiring: podman invocations need the lab user manager reachable — `XDG_RUNTIME_DIR=/run/user/<lab-uid>` and `DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/<lab-uid>/bus`. This re-opens ADR-0057's `XDG_RUNTIME_DIR=/run/lab` + `RuntimeDirectoryPreserve` decisions: decide whether podman's runroot/tmpdir move to `/run/user/<uid>` (linger keeps it alive, so `RuntimeDirectoryPreserve` machinery may become unnecessary) and document in the ADR. - **Delete the whole hand-rolled layout**: `lab-payload.service` holder, `lab-cgroup-hygiene` oneshot, `podmanx.setupCgroups`, the per-spawn restart-safety guard, and `Delegate`/`DelegateSubgroup` on `lab.service`. `lab.service` restarts can then never EBUSY, and containers survive lab restarts by construction (independent scope units). - **Preflight becomes a real spawn probe.** The current preflight was false-green: it verifies controllers/ownership but never performs an actual PID migration — the one operation that fails. Replace with: spawn a trivial container with `--memory` set, assert it starts and the cap landed in its scope cgroup. That catches this whole class regardless of layout (preserves #205's no-false-green property for real). Structural wins: each failure mode disappears rather than being pinned — no unit we manage carries delegation, `user@lab` is systemd's own machinery (never restarted by nixos-rebuild, and stopping it kills its whole subtree, so it has no survivor-EBUSY mode either). ## Migration / repo state notes - **ADR-0059 and the holder implementation are deployed on dev-new but are NOT on repo main** (repo main still has the ADR-0058 in-unit layout; `docs/adr/` ends at 0058). Locate that branch or supersede in place; the new ADR should take the next free number and note the phantom deployed "ADR-0059". - Survivor containers under the old cgroup locations keep running uncapped until natural exit — accepted, same stance as ADR-0058's migration note. Prune the ~1460 empty `libpod-*` leftovers under the holder's `payload/`, then the holder unit can be removed (its cgroup dies with its last survivor). - Emergency unblock currently available on dev-new until this lands (documented, applied at the maintainer's discretion): `chown lab /sys/fs/cgroup/system.slice/cgroup.procs` satisfies the ancestor check; holds until systemd re-realizes the slice. ## Acceptance 1. Provider login round-trip works (login pane container starts, `/auth/login/start` 200). 2. Agent run containers start with `--memory`/`--pids-limit` verifiably applied to their scope cgroup. 3. `systemctl restart lab` with live containers: lab comes back clean, containers survive, and a new container can be spawned afterwards. 4. Preflight fails loudly (actionable message) when the user manager is unreachable, instead of false-green. 5. `lab-payload.service`, `lab-cgroup-hygiene`, `setupCgroups`, and the spawn guard are gone from module + code. References: #237 (root-cause of the restart side, closed), ADR-0052, ADR-0057, ADR-0058 (fallback promoted by this issue), deployed-only "ADR-0059".

dominik.polakovics added the

ready-for-agent

bug

labels

2026-07-26 10:14:08 +02:00

dominik.polakovics referenced this issue from a commit

2026-07-26 10:54:19 +02:00

fix(podmanx,nix): containers as user@lab transient scopes — linger + systemd cgroup manager (ADR-0060)

dominik.polakovics referenced this issue from a pull request that will close it,

2026-07-26 10:54:42 +02:00

fix(podmanx,nix): containers as user@lab transient scopes — linger + systemd cgroup manager (ADR-0060) #240

dominik.polakovics referenced this issue from a commit