feat(runner): containerized runner — rootless podman panes behind repos.runner #214

Merged

dominik.polakovics merged 3 commits from afk/205 into main 2026-07-23 15:20:28 +02:00

Conversation 2 Commits 3 Files changed 81 +4738 -649

dominik.polakovics commented 2026-07-23 05:31:41 +02:00

Owner

Copy link

Closes #205

First end-to-end containerized run per the 2026-07-22 container-isolation design, now recorded as ADR-0052: tmux stays host-side and keeps owning liveness/attach/send-keys/capture; a repo set to runner = container gets its pane command wrapped as podman run -it --rm (rootless podman + crun) with the full mount inventory. Host mode stays the default and is labeled "unsandboxed — full host access" (break-glass) in the repo-settings UI.

What shipped

• repos.runner end to end — migration 0017 (sqlite+postgres), store, reposvc validation, PATCH API, and a new repo-settings Runner section (select + host warning + limit overrides with inherit hints).
• internal/podmanx — pure pane-argv construction (--userns=keep-id, --network=pasta, --mount type=image of the agent-tools image at /opt/lab, five binds, -w), deterministic labrun- container names (recomputable, no stored state), HOME env rewriting, the podman rm --force --ignore backstop, and the collect-all preflight (podman ≥ 4, pasta, subuid/subgid, cgroup-v2 memory+pids delegation, dev image + tools refs resolvable) over injectable probes.
• Limits — --memory/--pids-limit/--ulimit nofile from seeded global settings rows (8g/4096/16384) with nullable per-repo overrides; the prlimit wrapper is retired for container panes only (tmuxx.WithoutNofileCap); host panes byte-identical to before (dual-runner seam tested).
• Per-run runtime dir — <state>/instances/<runID>/runtime now holds the run's git cred files, known_hosts (TOFU-seeded from the global file), dialog spool, and --settings file, so the container mounts exactly one run's secret surface; wiped with the run (host mode uses the same path — one code path).
• Socket move — agent.sock → <state>/agent/agent.sock so the socket directory can be mounted (a socket-file bind pins a dead inode across a server restart); back-compat symlink at the old path keeps pre-upgrade sessions working.
• Env discipline — nothing inherited: non-secrets ride --env K=V; LAB_TOKEN rides tmux -e + name-only --env (never in any argv); LAB_URL pinned to the mounted unix socket (a TCP --agent-url is unreachable through pasta's no-host-route netns by design); PATH pins /opt/lab/bin first.
• Refusals & teardown — container spawns on an unready host are refused with the collected actionable preflight failures before the claim (no issue parked by a host problem); Stop/reap/Discard follow the tmux kill with the rm backstop; startup reconciliation sweeps orphaned labrun- containers.
• Pre-push hook: no change needed — it already invokes labctl by bare PATH lookup, and the container PATH carries /opt/lab/bin (ADR-0051's static labctl). No hook-marker migration.
• Config — --podman, --container-image (global dev image; empty ⇒ preflight refusal), --container-tools-image provider=ref[,…] (@sha256-pinned per ADR-0051).
• Docs — ADR-0052, CONTEXT.md Runner glossary entry, ops.md "Container runner" operator section (provisioning: subuid/subgid, Delegate=yes, XDG_RUNTIME_DIR/linger, passt), definition-of-done §11.

Verification

Automated, all green locally: go test ./... and go test -tags ui ./... (44 packages, incl. the afk cycle and instance integration suites against real tmux), golangci-lint run, go vet, vitest (930 tests / 64 files), tsc --noEmit, vite build, migration-parity tests. The pane argv, mounts, env split, limits resolution, refusal paths, stop backstops, and the full preflight matrix are unit-pinned (internal/podmanx, internal/instance/container_test.go, internal/reconcile/container_test.go, internal/afk/container_test.go).

Not claiming e2e: the lab sandbox has no rootless podman, so the real-podman integration test (TestContainerPane_integration, gated on LAB_TEST_PODMAN=1) skips here — verified to skip gracefully with distinct messages for each missing prerequisite. The acceptance criteria's full container loop (spawn → chat via tailer → labctl over the socket → push with hook scan → stop cleanup) and the in-container reachability assertions (state dir/master.key/host loopback unreachable; agent.sock + egress reachable) remain maintainer validation on the dev host, per the issue brief. definition-of-done §11 records exactly this split.

Notes for the dev-host pass

• Provision per ops.md § Container runner: podman ≥ 4 + crun + passt, subuid/subgid for the service user, Delegate=yes, XDG_RUNTIME_DIR for rootless podman, then set --container-image and --container-tools-image claude-code=<ref@sha256:…>.
• Startup logs the preflight verdict (version on OK, one warn per failure with hint); container spawns surface the same message as a 400.
• Upgrade note: the socket path moved; the symlink covers pre-upgrade sessions.
• Scope fences honored: containerized login (#206) and per-repo image_ref (#207) are untouched follow-ups; the default stays host until preflight is proven.

Closes #205 First end-to-end containerized run per the 2026-07-22 container-isolation design, now recorded as **ADR-0052**: tmux stays host-side and keeps owning liveness/attach/send-keys/capture; a repo set to `runner = container` gets its pane command wrapped as `podman run -it --rm` (rootless podman + crun) with the full mount inventory. Host mode stays the default and is labeled **"unsandboxed — full host access"** (break-glass) in the repo-settings UI. ## What shipped - **`repos.runner` end to end** — migration 0017 (sqlite+postgres), store, reposvc validation, PATCH API, and a new repo-settings **Runner** section (select + host warning + limit overrides with inherit hints). - **`internal/podmanx`** — pure pane-argv construction (`--userns=keep-id`, `--network=pasta`, `--mount type=image` of the agent-tools image at `/opt/lab`, five binds, `-w`), deterministic `labrun-` container names (recomputable, no stored state), HOME env rewriting, the `podman rm --force --ignore` backstop, and the collect-all **preflight** (podman ≥ 4, pasta, subuid/subgid, cgroup-v2 memory+pids delegation, dev image + tools refs resolvable) over injectable probes. - **Limits** — `--memory/--pids-limit/--ulimit nofile` from seeded global settings rows (`8g`/`4096`/`16384`) with nullable per-repo overrides; the prlimit wrapper is retired for container panes only (`tmuxx.WithoutNofileCap`); host panes byte-identical to before (dual-runner seam tested). - **Per-run runtime dir** — `<state>/instances/<runID>/runtime` now holds the run's git cred files, known_hosts (TOFU-seeded from the global file), dialog spool, and `--settings` file, so the container mounts exactly one run's secret surface; wiped with the run (host mode uses the same path — one code path). - **Socket move** — `agent.sock` → `<state>/agent/agent.sock` so the socket *directory* can be mounted (a socket-file bind pins a dead inode across a server restart); back-compat symlink at the old path keeps pre-upgrade sessions working. - **Env discipline** — nothing inherited: non-secrets ride `--env K=V`; `LAB_TOKEN` rides tmux `-e` + name-only `--env` (never in any argv); `LAB_URL` pinned to the mounted unix socket (a TCP `--agent-url` is unreachable through pasta's no-host-route netns by design); `PATH` pins `/opt/lab/bin` first. - **Refusals & teardown** — container spawns on an unready host are refused with the collected actionable preflight failures *before* the claim (no issue parked by a host problem); Stop/reap/Discard follow the tmux kill with the rm backstop; startup reconciliation sweeps orphaned `labrun-` containers. - **Pre-push hook: no change needed** — it already invokes `labctl` by bare PATH lookup, and the container PATH carries `/opt/lab/bin` (ADR-0051's static labctl). No hook-marker migration. - **Config** — `--podman`, `--container-image` (global dev image; empty ⇒ preflight refusal), `--container-tools-image provider=ref[,…]` (@sha256-pinned per ADR-0051). - **Docs** — ADR-0052, CONTEXT.md **Runner** glossary entry, ops.md "Container runner" operator section (provisioning: subuid/subgid, `Delegate=yes`, `XDG_RUNTIME_DIR`/linger, passt), definition-of-done §11. ## Verification Automated, all green locally: `go test ./...` and `go test -tags ui ./...` (44 packages, incl. the afk cycle and instance integration suites against real tmux), `golangci-lint run`, `go vet`, vitest (930 tests / 64 files), `tsc --noEmit`, vite build, migration-parity tests. The pane argv, mounts, env split, limits resolution, refusal paths, stop backstops, and the full preflight matrix are unit-pinned (`internal/podmanx`, `internal/instance/container_test.go`, `internal/reconcile/container_test.go`, `internal/afk/container_test.go`). **Not claiming e2e:** the lab sandbox has no rootless podman, so the real-podman integration test (`TestContainerPane_integration`, gated on `LAB_TEST_PODMAN=1`) skips here — verified to skip gracefully with distinct messages for each missing prerequisite. The acceptance criteria's full container loop (spawn → chat via tailer → labctl over the socket → push with hook scan → stop cleanup) and the in-container reachability assertions (state dir/master.key/host loopback unreachable; agent.sock + egress reachable) remain **maintainer validation on the dev host**, per the issue brief. definition-of-done §11 records exactly this split. ## Notes for the dev-host pass - Provision per ops.md § Container runner: podman ≥ 4 + crun + passt, subuid/subgid for the service user, `Delegate=yes`, `XDG_RUNTIME_DIR` for rootless podman, then set `--container-image` and `--container-tools-image claude-code=<ref@sha256:…>`. - Startup logs the preflight verdict (version on OK, one warn per failure with hint); container spawns surface the same message as a 400. - Upgrade note: the socket path moved; the symlink covers pre-upgrade sessions. - Scope fences honored: containerized login (#206) and per-repo `image_ref` (#207) are untouched follow-ups; the default stays `host` until preflight is proven.

dominik.polakovics added 1 commit 2026-07-23 05:31:42 +02:00

feat(runner): containerized runner — rootless podman panes behind repos.runner (#205) ... f446550998

The 2026-07-22 container-isolation tracer bullet (ADR-0052): tmux stays
host-side and keeps owning liveness/attach/capture, while a repo set to
runner=container gets its pane command wrapped as `podman run -it --rm`
with the full mount inventory — worktree + bare rw at host-identical
paths, the agent socket directory, the instance HOME at /home/agent, the
per-run runtime dir, and the ADR-0051 agent-tools image read-only at
/opt/lab — under --userns=keep-id and --network=pasta.

- repos.runner ('container'|'host', host default/break-glass, labeled
  "unsandboxed — full host access") end to end: migration 0017, store,
  PATCH validation, repo-settings Runner section.
- internal/podmanx: pane argv construction, deterministic labrun-
  container names, HOME env rewrite, rm backstop, and the collect-all
  startup preflight (podman>=4, pasta, subuid/subgid, cgroup-v2
  delegation, images resolvable) behind injectable probes.
- Limits --memory/--pids-limit/--ulimit nofile from seeded settings rows
  (8g/4096/16384) with nullable per-repo overrides; prlimit retired for
  container panes only (tmuxx WithoutNofileCap).
- Per-run runtime dir <state>/instances/<runID>/runtime for git creds,
  known_hosts (TOFU-seeded from the global file), dialog spool, and the
  --settings file — the container mounts exactly one run's secret
  surface; wiped with the run.
- agent.sock moved into <state>/agent/ so the socket DIRECTORY can be
  mounted (a socket-file bind pins a dead inode across restart);
  back-compat symlink kept at the old path.
- Env discipline: nothing inherited; non-secrets as --env K=V, LAB_TOKEN
  name-only forwarded from tmux -e; LAB_URL pinned to the mounted unix
  socket; PATH pins /opt/lab/bin first.
- Container spawns refused with actionable preflight errors BEFORE the
  claim; stop/reap/discard kill paths gained the podman rm backstop and
  startup reconciliation sweeps orphaned labrun- containers.
- Pre-push hook unchanged: it already PATH-resolves labctl, and the
  container PATH carries /opt/lab/bin.
- Integration test gated on LAB_TEST_PODMAN=1, skipping gracefully where
  rootless podman is unavailable; argv/preflight/seam unit-tested
  throughout. Docs: ADR-0052, CONTEXT.md Runner entry, ops.md container
  runner section, definition-of-done §11.

Closes #205

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics commented 2026-07-23 13:03:40 +02:00

Author

Owner

Copy link

[autoland] verdict: pass

This was generated by AI while landing a PR.

PASS — validated, landable as-is.

Signal relied on: Forgejo Actions CI green — ci / native (7m53s) and ci-nix / flake-check (39m12s) both success. Not re-run. Deep diff review of the security-critical paths (podmanx argv, launch/container/stop, reconcile, afk, agentapi, store, httpapi, web Runner).

Conventions: Conventional Commits title, valid Closes #205. Merges cleanly into main (no conflict).

Verified correct:

• LAB_TOKEN rides tmux -e + name-only --env LAB_TOKEN — never appears as a value in any podman argv (regression-pinned, container_test.go:135-144).
• Minimal 5-bind mount inventory (worktree, this-repo bare, agent socket dir, this-run home, this-run runtime) + RO tools image — no state root, master.key, or other repos exposed.
• Preflight returns a 400 refusal (not a warning) for every missing prerequisite, and runs before the claim so an AFK refusal parks no issue.
• podman rm --force --ignore backstops every kill path + startup orphan sweep.
• Host-mode dual-runner seam intact (untouched else branch; WithoutNofileCap is a zero-value StartOpt).

CONCERNS (non-blocking — already scoped as dev-host maintainer validation in the PR & ADR-0052):

1. Host-loopback unreachability rests on --network=pasta defaults with no CI coverage (integration test skips without podman). Confirm on the dev host that a container cannot reach a host 127.0.0.1 service.
2. Cgroup-delegation preflight reads the lab process's own /proc/self/cgroup; verify the deployed unit delegates to the subtree the pane's container actually lands in, else limits could be silently absent despite green preflight.

[autoland] verdict: pass > *This was generated by AI while landing a PR.* **PASS** — validated, landable as-is. **Signal relied on:** Forgejo Actions CI green — `ci / native` (7m53s) and `ci-nix / flake-check` (39m12s) both success. Not re-run. Deep diff review of the security-critical paths (podmanx argv, launch/container/stop, reconcile, afk, agentapi, store, httpapi, web Runner). **Conventions:** Conventional Commits title, valid `Closes #205`. Merges cleanly into main (no conflict). **Verified correct:** - LAB_TOKEN rides tmux `-e` + name-only `--env LAB_TOKEN` — never appears as a value in any podman argv (regression-pinned, container_test.go:135-144). - Minimal 5-bind mount inventory (worktree, this-repo bare, agent socket dir, this-run home, this-run runtime) + RO tools image — no state root, master.key, or other repos exposed. - Preflight returns a 400 refusal (not a warning) for every missing prerequisite, and runs before the claim so an AFK refusal parks no issue. - `podman rm --force --ignore` backstops every kill path + startup orphan sweep. - Host-mode dual-runner seam intact (untouched else branch; WithoutNofileCap is a zero-value StartOpt). **CONCERNS (non-blocking — already scoped as dev-host maintainer validation in the PR & ADR-0052):** 1. Host-loopback unreachability rests on `--network=pasta` defaults with no CI coverage (integration test skips without podman). Confirm on the dev host that a container cannot reach a host `127.0.0.1` service. 2. Cgroup-delegation preflight reads the lab process's own `/proc/self/cgroup`; verify the deployed unit delegates to the subtree the pane's container actually lands in, else limits could be silently absent despite green preflight.

dominik.polakovics added 1 commit 2026-07-23 13:50:20 +02:00

fix(podmanx): land container in lab's delegated cgroup and verify real delegation ... d05682871c

The container-runner preflight could pass on a host where a run's
--memory/--pids-limit caps would be silently absent — the exact "green
preflight, no limits" failure #205's blast-radius contract is meant to
prevent. Two coupled gaps:

- The delegation check read cgroup.controllers of lab's own cgroup. That
  file reflects the controllers the PARENT delegated inward, so it lists
  "memory pids" even on a unit with Delegate=no, where lab cannot create a
  single limited child cgroup. (subtree_control is no substitute — a
  correctly delegated but idle cgroup has it empty until the first child.)
- Rootless podman's default placed the container under user.slice, a
  subtree the preflight never inspects, so even a correct check on
  lab.service governed the wrong cgroup.

Fixes:
- RunArgv pins --cgroups=split, placing the payload cgroup inside lab's own
  cgroup (<lab's cgroup>/libpod-payload-<id>) rather than user.slice, so the
  caps apply in the exact subtree preflight checks. Split reuses lab's
  (delegated) cgroup for conmon+payload and requires Delegate=yes.
- Preflight now also requires lab's cgroup directory to be writable by the
  service user — the durable signal systemd's Delegate=yes leaves (it chowns
  the delegated cgroup dir to the unit user). Only write access distinguishes
  a delegated cgroup from a non-delegated one that merely inherited the
  controllers.

Verified on the dev host: lab.service (Delegate=no) lists "cpuset cpu io
memory pids" in cgroup.controllers but its cgroup dir is root-owned and
unwritable by lab — the old check passed, the new one correctly refuses with
an actionable Delegate=yes hint.

Also documents the pasta caveat: only loopback-bound host services are
unreachable from a container; wildcard/LAN-bound services (including lab's
own --addr :8080) stay reachable via host.containers.internal.

Refs #205

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

dominik.polakovics referenced this pull request 2026-07-23 13:50:56 +02:00

Container runner: wildcard-bound host services reachable from containers via host.containers.internal #216

dominik.polakovics scheduled this pull request to auto merge when all checks succeed 2026-07-23 14:11:05 +02:00

dominik.polakovics commented 2026-07-23 15:09:18 +02:00

Author

Owner

Copy link

[autoland] verdict: pass

This was generated by AI while landing a PR.

PASS — validated independently, landable as-is.

Signal relied on: Forgejo Actions CI green — ci / native (8m33s) and ci-nix / flake-check (21m15s) both success (aggregate success, exit 0). Not re-run. Plus a focused read of the security-critical paths: podmanx argv/env/mount/name construction, preflight, the launch dual-runner seam, container wiring, reconcile orphan sweep, migrations, config parsing, and the socket move.

Conventions: Conventional Commits title; valid Closes #205. Merges cleanly into current main (git merge-tree --write-tree, no conflict).

Verified correct:

• Secret split holds: containerEnv forwards only LAB_TOKEN by name (--env K), secretForwardEnv hands tmux -e exactly LAB_TOKEN; no secret value ever enters the podman argv (regression-pinned). LAB_URL is rewritten to the mounted unix socket.
• Mount inventory is minimal and host-path-identical (worktree, bare, agent socket dir, per-run home→/home/agent, per-run runtime) + RO tools image — no state root, master.key, or other repos exposed.
• Preflight collects all failures, refuses container spawns before the AFK claim (no issue parked by a host problem), and detects cgroup delegation by write-access (not just cgroup.controllers) — the subtle false-green guard is correct.
• --cgroups=split lands the payload in lab's delegated subtree so --memory/--pids-limit actually bind; matches what preflight checks.
• Host-mode path byte-identical to pre-#205 (untouched else branch; WithoutNofileCap is a zero-value StartOpt). podman rm --force --ignore backstops every kill path; startup sweep re-checks the labrun- prefix and honors the startguard snapshot-before-list ordering.
• Migration 0017 (sqlite+postgres) adds runner NOT NULL DEFAULT 'host' — existing repos unchanged, container opt-in.

CONCERNS (non-blocking — scoped as dev-host maintainer validation in the PR & ADR-0052):

1. Host-loopback unreachability rests on --network=pasta defaults with no CI coverage (integration test skips without podman). Confirm on the dev host that a container cannot reach a host 127.0.0.1 service.
2. Cgroup-delegation preflight reads the lab process's own /proc/self/cgroup; verify the deployed unit delegates to the subtree the pane's container actually lands in.

[autoland] verdict: pass > *This was generated by AI while landing a PR.* **PASS** — validated independently, landable as-is. **Signal relied on:** Forgejo Actions CI green — `ci / native` (8m33s) and `ci-nix / flake-check` (21m15s) both success (aggregate success, exit 0). Not re-run. Plus a focused read of the security-critical paths: podmanx argv/env/mount/name construction, preflight, the launch dual-runner seam, container wiring, reconcile orphan sweep, migrations, config parsing, and the socket move. **Conventions:** Conventional Commits title; valid `Closes #205`. Merges cleanly into current main (git merge-tree --write-tree, no conflict). **Verified correct:** - Secret split holds: containerEnv forwards only LAB_TOKEN by name (--env K), secretForwardEnv hands tmux -e exactly LAB_TOKEN; no secret value ever enters the podman argv (regression-pinned). LAB_URL is rewritten to the mounted unix socket. - Mount inventory is minimal and host-path-identical (worktree, bare, agent socket *dir*, per-run home→/home/agent, per-run runtime) + RO tools image — no state root, master.key, or other repos exposed. - Preflight collects all failures, refuses container spawns *before* the AFK claim (no issue parked by a host problem), and detects cgroup delegation by write-access (not just cgroup.controllers) — the subtle false-green guard is correct. - --cgroups=split lands the payload in lab's delegated subtree so --memory/--pids-limit actually bind; matches what preflight checks. - Host-mode path byte-identical to pre-#205 (untouched else branch; WithoutNofileCap is a zero-value StartOpt). podman rm --force --ignore backstops every kill path; startup sweep re-checks the labrun- prefix and honors the startguard snapshot-before-list ordering. - Migration 0017 (sqlite+postgres) adds runner NOT NULL DEFAULT 'host' — existing repos unchanged, container opt-in. **CONCERNS (non-blocking — scoped as dev-host maintainer validation in the PR & ADR-0052):** 1. Host-loopback unreachability rests on --network=pasta defaults with no CI coverage (integration test skips without podman). Confirm on the dev host that a container cannot reach a host 127.0.0.1 service. 2. Cgroup-delegation preflight reads the lab process's own /proc/self/cgroup; verify the deployed unit delegates to the subtree the pane's container actually lands in.

dominik.polakovics added 1 commit 2026-07-23 15:11:59 +02:00

Merge branch 'main' into afk/205 d54517a292

dominik.polakovics merged commit 8ed758e72c into main 2026-07-23 15:20:28 +02:00

dominik.polakovics deleted branch afk/205 2026-07-23 15:20:29 +02:00

dominik.polakovics referenced this pull request from a commit 2026-07-23 15:20:29 +02:00

Merge pull request 'feat(runner): containerized runner — rootless podman panes behind repos.runner' (#214) from afk/205 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is broken

  

chore

  

enhancement

New feature or improvement

  

needs-info

Waiting on reporter for more information

  

needs-triage

Maintainer needs to evaluate this issue

  

ready-for-agent

Fully specified, ready for an AFK agent

  

ready-for-human

Requires human implementation

  

wontfix

Will not be actioned

No labels

bug

chore

enhancement

needs-info

needs-triage

ready-for-agent

ready-for-human

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Cloonar/coding-lab!214

No description provided.