Cloonar/coding-lab

Fork 0

feat(nix): first-class container-runner support in the NixOS module — services.lab.container.* #218

New issue

Closed

opened 2026-07-23 18:19:41 +02:00 by dominik.polakovics · 2 comments

dominik.polakovics commented

2026-07-23 18:19:41 +02:00

Owner

What to build

A services.lab.container submodule in the NixOS module, giving container-runner deployments (ADR-0052 / ADR-0053) first-class options instead of hand-wired extraFlags / extraPackages / raw systemd.services.lab.serviceConfig overrides. Today the module has no container support at all: an operator must assemble the flags, Delegate=yes, podman/passt on the unit PATH, subuid/subgid ranges, and the rootless runtime dir by hand from docs/ops.md.

Design grilled 2026-07-23; the decisions below are the contract.

Option surface

services.lab.container = {
  enable = true;                       # explicit switch gating ALL host mutations
  toolsImages = {                      # attrsOf nonEmptyStr, keyed by provider ID
    "claude-code" = "git.cloonar.com/cloonar/agent-tools@sha256:…";
    codex        = "git.cloonar.com/cloonar/agent-tools@sha256:…";
  };
  defaultImage = "debian:stable-slim"; # nullable — per-repo image_ref alone is valid (ADR-0053)
  subIdRange = { start = 100000; count = 65536; }; # nullable opt-out, overridable on collision
};

Rendered flags: --container-tools-image claude-code=ref[,codex=ref…] (comma-joined) and --container-image <defaultImage> when set, appended to the module's args before extraFlags.

Host provisioning (all gated on `container.enable`)

virtualisation.podman.enable = true (pulls in policy.json / registries.conf / crun via virtualisation.containers), plus podman and passt on the lab unit PATH so preflight's PATH lookup and the pane argv resolve.
serviceConfig.Delegate = true — delegate all controllers, matching the ops.md guidance the running deployment was validated against.
users.users.<lab user>.subUidRanges/subGidRanges from subIdRange — NixOS merges user attrs, so this works for operator-brought users too.
RuntimeDirectory = "lab" + RuntimeDirectoryPreserve = "yes" + XDG_RUNTIME_DIR=/run/lab in the unit environment. Preserve is load-bearing: KillMode=process keeps podman-attached containers alive across a lab restart/deploy, and a plain RuntimeDirectory would be wiped underneath them, destroying rootless podman's runtime state (pause-process refs, netns bookkeeping).

Eval-time guardrails

Assertion: container.enable → toolsImages != { } (the seedUser precedent: catch the typo'd deploy at rebuild, not after the service restarted).
No assertion on defaultImage (blank + per-repo refs is a valid deployment).
No eval-time validation of toolsImages keys against provider IDs (the agentPackages precedent; the server's boot error names the registered IDs, and a nix-side list would drift when new providers land, e.g. #126).
No digest-pin enforcement on refs (documented, not enforced — local tags are fine during bring-up).

Ships with

docs/ops.md "Container runner" section: NixOS guidance collapses to services.lab.container.*; add a note that rootless image storage lands under $HOME/.local/share/containers = the state dir, so dev images count toward stateDir disk sizing.
A flake check that builds a NixOS system closure with container.enable = true and dummy refs, so option typos and assertion regressions fail CI without booting anything.

Acceptance criteria

container.enable = true with toolsImages renders --container-tools-image provider=ref[,provider=ref…] (and --container-image when defaultImage is set) into ExecStart
Enabling container mode provisions everything preflight checks: virtualisation.podman.enable, podman+passt on the unit PATH, Delegate=true, subuid/subgid ranges from subIdRange, and RuntimeDirectory=lab with RuntimeDirectoryPreserve=yes + XDG_RUNTIME_DIR exported
Eval assertion fires on container.enable = true with empty toolsImages; defaultImage = null deploys fine
subIdRange = null opts out of user-range provisioning; a custom start/count is honored
With container.enable = false (the default) the generated unit is unchanged from today — host-mode deployments see zero diff
Flake check builds a container-enabled NixOS system closure with dummy refs in CI
docs/ops.md NixOS guidance rewritten to services.lab.container.*, including the stateDir image-storage sizing note
Maintainer, at landing (not part of the agent's DoD): validated on a real host — preflight green after one rebuild, a container-runner repo spawns, and running containers survive a lab restart with their runtime dir intact

## What to build A `services.lab.container` submodule in the NixOS module, giving container-runner deployments (ADR-0052 / ADR-0053) first-class options instead of hand-wired `extraFlags` / `extraPackages` / raw `systemd.services.lab.serviceConfig` overrides. Today the module has no container support at all: an operator must assemble the flags, `Delegate=yes`, podman/passt on the unit PATH, subuid/subgid ranges, and the rootless runtime dir by hand from docs/ops.md. Design grilled 2026-07-23; the decisions below are the contract. ### Option surface ```nix services.lab.container = { enable = true; # explicit switch gating ALL host mutations toolsImages = { # attrsOf nonEmptyStr, keyed by provider ID "claude-code" = "git.cloonar.com/cloonar/agent-tools@sha256:…"; codex = "git.cloonar.com/cloonar/agent-tools@sha256:…"; }; defaultImage = "debian:stable-slim"; # nullable — per-repo image_ref alone is valid (ADR-0053) subIdRange = { start = 100000; count = 65536; }; # nullable opt-out, overridable on collision }; ``` Rendered flags: `--container-tools-image claude-code=ref[,codex=ref…]` (comma-joined) and `--container-image <defaultImage>` when set, appended to the module's args before `extraFlags`. ### Host provisioning (all gated on `container.enable`) - `virtualisation.podman.enable = true` (pulls in policy.json / registries.conf / crun via `virtualisation.containers`), plus podman and passt on the lab unit PATH so preflight's PATH lookup and the pane argv resolve. - `serviceConfig.Delegate = true` — delegate all controllers, matching the ops.md guidance the running deployment was validated against. - `users.users.<lab user>.subUidRanges`/`subGidRanges` from `subIdRange` — NixOS merges user attrs, so this works for operator-brought users too. - `RuntimeDirectory = "lab"` + `RuntimeDirectoryPreserve = "yes"` + `XDG_RUNTIME_DIR=/run/lab` in the unit environment. **Preserve is load-bearing:** `KillMode=process` keeps podman-attached containers alive across a lab restart/deploy, and a plain RuntimeDirectory would be wiped underneath them, destroying rootless podman's runtime state (pause-process refs, netns bookkeeping). ### Eval-time guardrails - Assertion: `container.enable` → `toolsImages != { }` (the seedUser precedent: catch the typo'd deploy at rebuild, not after the service restarted). - No assertion on `defaultImage` (blank + per-repo refs is a valid deployment). - No eval-time validation of `toolsImages` keys against provider IDs (the agentPackages precedent; the server's boot error names the registered IDs, and a nix-side list would drift when new providers land, e.g. #126). - No digest-pin enforcement on refs (documented, not enforced — local tags are fine during bring-up). ### Ships with - docs/ops.md "Container runner" section: NixOS guidance collapses to `services.lab.container.*`; add a note that rootless image storage lands under `$HOME/.local/share/containers` = the state dir, so dev images count toward stateDir disk sizing. - A flake check that builds a NixOS system closure with `container.enable = true` and dummy refs, so option typos and assertion regressions fail CI without booting anything. ## Acceptance criteria - [ ] `container.enable = true` with toolsImages renders `--container-tools-image provider=ref[,provider=ref…]` (and `--container-image` when `defaultImage` is set) into ExecStart - [ ] Enabling container mode provisions everything preflight checks: `virtualisation.podman.enable`, podman+passt on the unit PATH, `Delegate=true`, subuid/subgid ranges from `subIdRange`, and `RuntimeDirectory=lab` with `RuntimeDirectoryPreserve=yes` + `XDG_RUNTIME_DIR` exported - [ ] Eval assertion fires on `container.enable = true` with empty `toolsImages`; `defaultImage = null` deploys fine - [ ] `subIdRange = null` opts out of user-range provisioning; a custom start/count is honored - [ ] With `container.enable = false` (the default) the generated unit is unchanged from today — host-mode deployments see zero diff - [ ] Flake check builds a container-enabled NixOS system closure with dummy refs in CI - [ ] docs/ops.md NixOS guidance rewritten to `services.lab.container.*`, including the stateDir image-storage sizing note - [ ] **Maintainer, at landing (not part of the agent's DoD):** validated on a real host — preflight green after one rebuild, a container-runner repo spawns, and running containers survive a lab restart with their runtime dir intact

dominik.polakovics added the

enhancement

label

2026-07-23 18:19:41 +02:00

dominik.polakovics commented

2026-07-23 18:20:03 +02:00

Author

Owner

This was generated by AI during triage.

Human Brief

Category: enhancement
Summary: Add a services.lab.container submodule to the NixOS module that provisions everything the container runner's preflight checks — flags, podman/passt, cgroup delegation, subuid/subgid ranges, and the rootless runtime dir — behind one explicit enable switch.

The issue body's "What to build" and acceptance criteria are the authoritative spec (grilled 2026-07-23). Not blocked: the container runner (#205/#214) and per-repo dev images (#207/#217) are merged; this is pure deployment surface.

Why ready-for-human rather than ready-for-agent:
The CI-checkable half (option wiring, flag rendering, the assertion, the closure-build flake check) an agent could ship. But the load-bearing decisions only prove out on a real NixOS host: that Delegate=true yields a preflight-green delegated cgroup, that RuntimeDirectoryPreserve=yes actually keeps live containers working across a lab restart (the KillMode=process invariant this design exists to protect), and that the subuid merge behaves on an operator-brought user. The agent sandbox can run neither podman nor nixos-rebuild — same posture as #206: an agent would ship the risky parts blind, and a wrong serviceConfig bit surfaces only as a broken deploy later.

Key interfaces (for whoever picks it up):

New options: services.lab.container.{enable, toolsImages, defaultImage, subIdRange} — shapes and semantics per the issue body; all host mutations gated on enable, never on option non-emptiness.
Flag rendering joins the module's existing args list before extraFlags, so an operator's hand-rolled container flags (today's workaround) still win during migration.
NixOS surface touched when enabled: virtualisation.podman.enable, the unit path, serviceConfig.Delegate, serviceConfig.RuntimeDirectory{,Preserve,Mode}, unit env XDG_RUNTIME_DIR, users.users.<user>.sub{Uid,Gid}Ranges.
The server side needs no changes — cmd/lab already wires container mode from the flags, and preflight remains the runtime authority on host readiness.

Acceptance criteria: the checklist in the issue body.

Out of scope:

Containerized provider login (#206) — hosts still need the provider CLIs for the login flow.
A NixOS VM test booting podman with stub images (deliberately rejected during grilling as heavyweight for host-specific risk; the flake check is eval/build-only).
The container network-isolation gap (#216) and any server-side flag or preflight changes.
Changing the rootless storage location (images land under stateDir by design; documented, not made configurable).

> *This was generated by AI during triage.* ## Human Brief **Category:** enhancement **Summary:** Add a `services.lab.container` submodule to the NixOS module that provisions everything the container runner's preflight checks — flags, podman/passt, cgroup delegation, subuid/subgid ranges, and the rootless runtime dir — behind one explicit `enable` switch. The issue body's "What to build" and acceptance criteria are the authoritative spec (grilled 2026-07-23). Not blocked: the container runner (#205/#214) and per-repo dev images (#207/#217) are merged; this is pure deployment surface. **Why ready-for-human rather than ready-for-agent:** The CI-checkable half (option wiring, flag rendering, the assertion, the closure-build flake check) an agent could ship. But the load-bearing decisions only prove out on a real NixOS host: that `Delegate=true` yields a preflight-green delegated cgroup, that `RuntimeDirectoryPreserve=yes` actually keeps live containers working across a lab restart (the KillMode=process invariant this design exists to protect), and that the subuid merge behaves on an operator-brought user. The agent sandbox can run neither podman nor `nixos-rebuild` — same posture as #206: an agent would ship the risky parts blind, and a wrong `serviceConfig` bit surfaces only as a broken deploy later. **Key interfaces (for whoever picks it up):** - New options: `services.lab.container.{enable, toolsImages, defaultImage, subIdRange}` — shapes and semantics per the issue body; all host mutations gated on `enable`, never on option non-emptiness. - Flag rendering joins the module's existing args list *before* `extraFlags`, so an operator's hand-rolled container flags (today's workaround) still win during migration. - NixOS surface touched when enabled: `virtualisation.podman.enable`, the unit `path`, `serviceConfig.Delegate`, `serviceConfig.RuntimeDirectory{,Preserve,Mode}`, unit env `XDG_RUNTIME_DIR`, `users.users.<user>.sub{Uid,Gid}Ranges`. - The server side needs **no** changes — `cmd/lab` already wires container mode from the flags, and preflight remains the runtime authority on host readiness. **Acceptance criteria:** the checklist in the issue body. **Out of scope:** - Containerized provider login (#206) — hosts still need the provider CLIs for the login flow. - A NixOS VM test booting podman with stub images (deliberately rejected during grilling as heavyweight for host-specific risk; the flake check is eval/build-only). - The container network-isolation gap (#216) and any server-side flag or preflight changes. - Changing the rootless storage location (images land under stateDir by design; documented, not made configurable).

dominik.polakovics added the

ready-for-human

label

2026-07-23 18:20:03 +02:00

dominik.polakovics commented

2026-07-23 18:22:26 +02:00

Author

Owner

This was generated by AI during triage.

Brief addendum — re-labeled `ready-for-agent` (maintainer override)

The Human Brief above stands as the contract, with one re-scope: this is agent-implementable after all. Unlike #206 (where every acceptance criterion required a live host loop), the CI-checkable half of this change — option wiring, flag rendering, the eval assertion, and the closure-build flake check — is fully verifiable by an agent. The host-behavioral criteria are validated once, by the maintainer, at landing on the dev host, which happens during deploy anyway.

For the implementing agent:

Your Definition of Done is every acceptance criterion in the issue body except the last one, which is explicitly marked as the maintainer's landing gate.
The flake check (NixOS system closure with container.enable = true and dummy refs) is your primary verification vehicle — make it exercise the assertion path too (a closure with empty toolsImages must fail eval).
Also verify the zero-diff criterion mechanically: the rendered unit for a container.enable = false config must be unchanged from today's module output.
The "Why ready-for-human" section above is superseded by this comment; the "Key interfaces" and "Out of scope" sections still apply verbatim.

> *This was generated by AI during triage.* ## Brief addendum — re-labeled `ready-for-agent` (maintainer override) The Human Brief above stands as the contract, with one re-scope: this is agent-implementable after all. Unlike #206 (where *every* acceptance criterion required a live host loop), the CI-checkable half of this change — option wiring, flag rendering, the eval assertion, and the closure-build flake check — is fully verifiable by an agent. The host-behavioral criteria are validated **once, by the maintainer, at landing** on the dev host, which happens during deploy anyway. For the implementing agent: - Your Definition of Done is every acceptance criterion in the issue body **except** the last one, which is explicitly marked as the maintainer's landing gate. - The flake check (NixOS system closure with `container.enable = true` and dummy refs) is your primary verification vehicle — make it exercise the assertion path too (a closure with empty `toolsImages` must fail eval). - Also verify the zero-diff criterion mechanically: the rendered unit for a `container.enable = false` config must be unchanged from today's module output. - The "Why ready-for-human" section above is superseded by this comment; the "Key interfaces" and "Out of scope" sections still apply verbatim.

dominik.polakovics added

ready-for-agent

and removed

ready-for-human

labels

2026-07-23 18:22:26 +02:00

dominik.polakovics referenced this issue from a pull request that will close it,

2026-07-23 19:25:15 +02:00

feat(nix): first-class container-runner support — services.lab.container.* #219

dominik.polakovics referenced this issue from a pull request that will close it,

2026-07-23 19:45:03 +02:00

feat(nix): first-class container-runner support — services.lab.container.* #219