fix(nix,podmanx): move payload cgroup delegation out of lab.service — DelegateSubgroup does not protect the main-process spawn, restart-EBUSY is back #237

New issue

Closed

opened 2026-07-26 00:34:40 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-07-26 00:34:40 +02:00

Owner

This was generated by AI during triage.

Incident

2026-07-26, dev-new, systemd 260.2: every lab.service restart after the ADR-0058 deploy failed with Failed to spawn executor: Device or resource busy (result resources), restart-looping past counter 700 while agent containers, tmux, conmon and pasta survived as designed. The on-disk layout was exactly what ADR-0058 specifies (unit root: no procs, controllers in subtree_control; main/: procs, clean; payload/: delegating, clean libpod-* leaves) and lab-cgroup-hygiene ran green before every attempt. Recovered manually by clearing payload/ then the unit root's subtree_control bottom-up (surviving containers run uncapped until they exit); the trap re-arms on every restart until this issue lands.

Root cause (verified in systemd v260 source; unchanged on master)

DelegateSubgroup= does not apply to the main process's spawn. In src/core/execute.c:

int exec_params_needs_control_subcgroup(const ExecParameters *params) {
        /* Keep this in sync with exec_params_get_cgroup_path(). */
        return FLAGS_SET(params->flags, EXEC_CGROUP_DELEGATE|EXEC_CONTROL_CGROUP|EXEC_IS_CONTROL);
}

All three flags are required; the main ExecStart process is not a control process, so exec_spawn() passes the unit cgroup root as the clone3(CLONE_INTO_CGROUP) target — sd-executor only migrates itself into the DelegateSubgroup afterwards. posix_spawn_wrapper retries without the cgroup flag only on ENOTSUP/EPERM, not EBUSY. Consequence: once lab has enabled controllers on the unit root's subtree_control (required to carry the payload caps) and anything survives a stop (KillMode=process is the product feature), the kernel's no-internal-process rule rejects every subsequent spawn. Fresh starts work only because the root's subtree_control is empty at spawn time — which is why ADR-0058 validated green and then failed on the next deploy.

ADR-0058's core placement ("systemd spawns lab into main/") is therefore wrong, and any design that keeps controllers enabled on lab.service's own cgroup root across restarts is structurally dead. The payload delegation must move out of lab.service's cgroup.

Sources: systemd v260 execute.c, systemd master execute.c, systemd v260 process-util.c.

Decided fix direction (maintainer-approved)

A separate, never-restarted holder unit owns the delegated payload cgroup; lab.service stops delegating entirely.

Agent Brief

Category: bug
Summary: Containers move to a delegated cgroup owned by a static holder unit (lab-payload.service); lab.service drops Delegate/DelegateSubgroup so its restarts can never hit the no-internal-process rule again.

Current behavior:
The NixOS module gives lab.service Delegate=yes + DelegateSubgroup=main; lab's preflight (setupCgroups) derives its own unit cgroup from /proc/self/cgroup, verifies it sits in a delegated subgroup (cgroup-layout check), enables +memory +pids on the unit root's subtree_control, and creates the payload/ sibling that --cgroup-parent pins for every container (session, AFK, login). A root lab-cgroup-hygiene oneshot clears stale controller delegation from main/ before each start. Result: first start works, every restart with survivors fails with EBUSY (see root cause above).

Desired behavior:

The NixOS module ships an additional static holder unit (working name lab-payload.service): Delegate=yes, DelegateSubgroup=main (its own trivial main process must sit in a subgroup so the root may delegate), KillMode=process (a holder stop must never take the containers down with it), same service user as lab, an ExecStart that just sleeps forever, and it must be excluded from deploy-driven restarts (restartIfChanged = false) — it never restarts, so the executor-spawn trap can never fire for it. systemd's Delegate=yes + User= chowns its cgroup subtree to the lab user, which is what lets rootless podman create the libpod-* children there.
lab.service loses Delegate and DelegateSubgroup entirely and gains Wants=/After= on the holder. Its own cgroup must never carry entries in subtree_control. KillMode=process, RuntimeDirectory/RuntimeDirectoryPreserve stay as they are.
lab's cgroup setup no longer derives the payload parent from its own cgroup: the payload cgroup lives under the holder unit's delegated cgroup (e.g. /system.slice/lab-payload.service/payload), and that is what Result.CgroupParent() / RunSpec.CgroupParent pin via --cgroup-parent for all container shapes (sessions, AFK runs, provider-login panes).
Preflight checks retarget: delegation/write-access and memory+pids verification move to the holder's cgroup, with an actionable failure hint naming the holder unit when it is missing, not running, or not delegated. The cgroup-layout check (lab-in-a-delegated-subgroup) is obsolete in its current form; the restart-safety guard's job becomes asserting that lab's own unit cgroup root has an empty subtree_control (nothing may ever re-arm the trap) alongside whatever holder-side invariants the implementation needs.
The lab-cgroup-hygiene oneshot is retargeted or retired: with lab.service undelegated, its cgroup is root-owned and rootless podman cannot dirty it; if a hygiene guard is kept, it belongs to the holder's main/ subgroup.
ADR: a new ADR supersedes/amends ADR-0058, documenting why DelegateSubgroup= cannot protect the main-process spawn (with the systemd source receipt) and recording the holder-unit design and its constraints (never restarted, KillMode=process, DelegateSubgroup on the holder).

Key interfaces:

The podmanx preflight Deps/setupCgroups/Cgroups surface — payload parent path derivation changes from self-cgroup-relative to holder-unit-cgroup; Verify() per-spawn guard keeps existing hard-fail semantics against the new invariants.
Result.CgroupParent() — same contract, new value; all three RunSpec shapes must keep pinning it.
Preflight check identifiers (cgroup-layout, cgroup-delegation, cgroup-restart-safety) — keep stable names where semantics survive, rename/retire where they do not; failure hints must name the holder unit.
NixOS module option surface — no new user-facing options required; holder unit is an implementation detail of container.enable.

Acceptance criteria:

With live containers + tmux survivors, systemctl restart lab succeeds repeatedly (no EBUSY, no manual surgery) — the exact dev-new incident scenario.
lab.service unit file contains no Delegate/DelegateSubgroup; its cgroup's subtree_control stays empty through container spawns and restarts.
Containers land under the holder's payload/ with --memory/--pids-limit caps observable in their libpod-* cgroups, and surviving containers keep their caps across lab restarts and deploys.
The holder unit is not restarted by a config/package change to lab itself (deploy simulation), and stopping the holder does not kill running containers (KillMode=process).
Preflight fails with an actionable holder-naming hint when the holder unit is absent/not delegated; unit tests cover the new layout logic against fake Deps (including the re-armed-trap detection on lab's own root).
New/amended ADR landed; ADR-0058 marked superseded/amended accordingly.

Out of scope:

Switching podman to the systemd cgroup manager / user-session (linger + user@) architecture — documented fallback in ADR-0058, still rejected.
Patching or pinning systemd; filing the upstream systemd issue is a nice-to-have, not part of this fix.
Re-applying lost caps to containers that survived the incident uncapped.
Changing cap values, container images, or the login-pane flow beyond the inherited --cgroup-parent value.

Validation note: full end-to-end restart validation needs a real NixOS host (dev-new deploy by the maintainer) — CI covers the Go layout/preflight logic and nix eval only.

> *This was generated by AI during triage.* ## Incident 2026-07-26, dev-new, systemd 260.2: every `lab.service` restart after the ADR-0058 deploy failed with `Failed to spawn executor: Device or resource busy` (result `resources`), restart-looping past counter 700 while agent containers, tmux, conmon and pasta survived as designed. The on-disk layout was **exactly** what ADR-0058 specifies (unit root: no procs, controllers in `subtree_control`; `main/`: procs, clean; `payload/`: delegating, clean `libpod-*` leaves) and `lab-cgroup-hygiene` ran green before every attempt. Recovered manually by clearing `payload/` then the unit root's `subtree_control` bottom-up (surviving containers run uncapped until they exit); the trap re-arms on every restart until this issue lands. ## Root cause (verified in systemd v260 source; unchanged on master) `DelegateSubgroup=` does **not** apply to the main process's spawn. In `src/core/execute.c`: ```c int exec_params_needs_control_subcgroup(const ExecParameters *params) { /* Keep this in sync with exec_params_get_cgroup_path(). */ return FLAGS_SET(params->flags, EXEC_CGROUP_DELEGATE|EXEC_CONTROL_CGROUP|EXEC_IS_CONTROL); } ``` All three flags are required; the main `ExecStart` process is not a control process, so `exec_spawn()` passes the **unit cgroup root** as the clone3(`CLONE_INTO_CGROUP`) target — sd-executor only migrates itself into the `DelegateSubgroup` afterwards. `posix_spawn_wrapper` retries without the cgroup flag only on ENOTSUP/EPERM, not EBUSY. Consequence: once lab has enabled controllers on the unit root's `subtree_control` (required to carry the payload caps) and *anything* survives a stop (KillMode=process is the product feature), the kernel's no-internal-process rule rejects every subsequent spawn. Fresh starts work only because the root's `subtree_control` is empty at spawn time — which is why ADR-0058 validated green and then failed on the next deploy. ADR-0058's core placement ("systemd spawns lab into `main/`") is therefore wrong, and **any design that keeps controllers enabled on lab.service's own cgroup root across restarts is structurally dead.** The payload delegation must move out of lab.service's cgroup. Sources: [systemd v260 execute.c](https://raw.githubusercontent.com/systemd/systemd/v260/src/core/execute.c), [systemd master execute.c](https://raw.githubusercontent.com/systemd/systemd/main/src/core/execute.c), [systemd v260 process-util.c](https://raw.githubusercontent.com/systemd/systemd/v260/src/basic/process-util.c). ## Decided fix direction (maintainer-approved) A separate, never-restarted **holder unit** owns the delegated payload cgroup; `lab.service` stops delegating entirely. ## Agent Brief **Category:** bug **Summary:** Containers move to a delegated cgroup owned by a static holder unit (`lab-payload.service`); `lab.service` drops `Delegate`/`DelegateSubgroup` so its restarts can never hit the no-internal-process rule again. **Current behavior:** The NixOS module gives `lab.service` `Delegate=yes` + `DelegateSubgroup=main`; lab's preflight (`setupCgroups`) derives its own unit cgroup from `/proc/self/cgroup`, verifies it sits in a delegated subgroup (`cgroup-layout` check), enables `+memory +pids` on the unit root's `subtree_control`, and creates the `payload/` sibling that `--cgroup-parent` pins for every container (session, AFK, login). A root `lab-cgroup-hygiene` oneshot clears stale controller delegation from `main/` before each start. Result: first start works, every restart with survivors fails with EBUSY (see root cause above). **Desired behavior:** - The NixOS module ships an additional static holder unit (working name `lab-payload.service`): `Delegate=yes`, `DelegateSubgroup=main` (its own trivial main process must sit in a subgroup so the root may delegate), `KillMode=process` (a holder stop must never take the containers down with it), same service user as lab, an `ExecStart` that just sleeps forever, and it must be excluded from deploy-driven restarts (`restartIfChanged = false`) — it never restarts, so the executor-spawn trap can never fire for it. systemd's `Delegate=yes` + `User=` chowns its cgroup subtree to the lab user, which is what lets rootless podman create the `libpod-*` children there. - `lab.service` loses `Delegate` and `DelegateSubgroup` entirely and gains `Wants=`/`After=` on the holder. Its own cgroup must never carry entries in `subtree_control`. `KillMode=process`, `RuntimeDirectory`/`RuntimeDirectoryPreserve` stay as they are. - lab's cgroup setup no longer derives the payload parent from its own cgroup: the payload cgroup lives under the holder unit's delegated cgroup (e.g. `/system.slice/lab-payload.service/payload`), and that is what `Result.CgroupParent()` / `RunSpec.CgroupParent` pin via `--cgroup-parent` for all container shapes (sessions, AFK runs, provider-login panes). - Preflight checks retarget: delegation/write-access and memory+pids verification move to the holder's cgroup, with an actionable failure hint naming the holder unit when it is missing, not running, or not delegated. The `cgroup-layout` check (lab-in-a-delegated-subgroup) is obsolete in its current form; the restart-safety guard's job becomes asserting that **lab's own unit cgroup root has an empty `subtree_control`** (nothing may ever re-arm the trap) alongside whatever holder-side invariants the implementation needs. - The `lab-cgroup-hygiene` oneshot is retargeted or retired: with lab.service undelegated, its cgroup is root-owned and rootless podman cannot dirty it; if a hygiene guard is kept, it belongs to the holder's `main/` subgroup. - ADR: a new ADR supersedes/amends ADR-0058, documenting why `DelegateSubgroup=` cannot protect the main-process spawn (with the systemd source receipt) and recording the holder-unit design and its constraints (never restarted, KillMode=process, DelegateSubgroup on the holder). **Key interfaces:** - The podmanx preflight `Deps`/`setupCgroups`/`Cgroups` surface — payload parent path derivation changes from self-cgroup-relative to holder-unit-cgroup; `Verify()` per-spawn guard keeps existing hard-fail semantics against the new invariants. - `Result.CgroupParent()` — same contract, new value; all three RunSpec shapes must keep pinning it. - Preflight check identifiers (`cgroup-layout`, `cgroup-delegation`, `cgroup-restart-safety`) — keep stable names where semantics survive, rename/retire where they do not; failure hints must name the holder unit. - NixOS module option surface — no new user-facing options required; holder unit is an implementation detail of `container.enable`. **Acceptance criteria:** - [ ] With live containers + tmux survivors, `systemctl restart lab` succeeds repeatedly (no EBUSY, no manual surgery) — the exact dev-new incident scenario. - [ ] `lab.service` unit file contains no `Delegate`/`DelegateSubgroup`; its cgroup's `subtree_control` stays empty through container spawns and restarts. - [ ] Containers land under the holder's `payload/` with `--memory`/`--pids-limit` caps observable in their `libpod-*` cgroups, and surviving containers keep their caps across lab restarts and deploys. - [ ] The holder unit is not restarted by a config/package change to lab itself (deploy simulation), and stopping the holder does not kill running containers (`KillMode=process`). - [ ] Preflight fails with an actionable holder-naming hint when the holder unit is absent/not delegated; unit tests cover the new layout logic against fake `Deps` (including the re-armed-trap detection on lab's own root). - [ ] New/amended ADR landed; ADR-0058 marked superseded/amended accordingly. **Out of scope:** - Switching podman to the systemd cgroup manager / user-session (linger + user@) architecture — documented fallback in ADR-0058, still rejected. - Patching or pinning systemd; filing the upstream systemd issue is a nice-to-have, not part of this fix. - Re-applying lost caps to containers that survived the incident uncapped. - Changing cap values, container images, or the login-pane flow beyond the inherited `--cgroup-parent` value. **Validation note:** full end-to-end restart validation needs a real NixOS host (dev-new deploy by the maintainer) — CI covers the Go layout/preflight logic and nix eval only.

dominik.polakovics added the

ready-for-agent

bug

labels

2026-07-26 00:34:40 +02:00

dominik.polakovics referenced this issue from a commit

2026-07-26 00:57:33 +02:00

fix(podmanx,nix): move payload cgroup delegation to a never-restarted holder unit (ADR-0059)

dominik.polakovics referenced this issue from a pull request that will close it,

2026-07-26 00:57:52 +02:00

fix(podmanx,nix): move payload cgroup delegation to a never-restarted lab-payload.service holder (ADR-0059) #238

dominik.polakovics closed this issue