Container-ready by default: container.enable=true, rev-pinned agent-tools release tags, verified pulls #221

Merged

dominik.polakovics merged 2 commits from lab/20260723-2144 into main 2026-07-24 12:09:25 +02:00

Conversation 2 Commits 2 Files changed 11 +488 -103

dominik.polakovics commented 2026-07-24 00:49:05 +02:00

Owner

Copy link

Closes #220

Diagnosing the dev-new rollout showed a pin bump alone can never activate the container runner: the enable switch lives in the host-config repo, and nothing tied an agent-tools image (whose baked-in labctl talks to this server) to the server's code rev. This PR makes the defaults carry the whole story, per ADR-0054 (new):

nix/module.nix

• services.lab.container.enable defaults true — services.lab.enable alone makes the host container-ready. enable = false is the explicit host-only opt-out and renders the byte-identical pre-container unit.
• New container.toolsImageRepo (default git.cloonar.com/cloonar/agent-tools); container.toolsImages defaults to <repo>:claude-<self.rev> / <repo>:codex-<self.rev>, falling back to the versions.env CLI-version tags when the source has no rev (dirty tree), so local nix flake check/rebuilds keep working.

flake.nix checks

• Zero-diff guard moved to an explicit enable = false dummy (podman leak-check added); new asserts pin default-on provisioning: tools flag with both default refs, no --container-image, Delegate=true, RuntimeDirectory=lab, podman+passt on the unit PATH, default subid range with no container config at all; virtualisation.podman on by default; explicitly-emptied toolsImages still fails eval.

.forgejo/workflows/agent-tools.yml

• Publish leg runs on every push to main (path gate kept on the PR smoke leg only) and pushes <provider>-<git sha> tags alongside the CLI-version tags.
• New verify step: logout, pull the sha tags into a scratch storage root, compare digests — anonymous pull is the deployment contract (the module provisions no registry credentials), so a package flipped private fails the publish loudly instead of failing every host's preflight quietly.

Go

• podmanx.Result.HasPullFailure() + a retry loop in cmd/lab: deploy and agent-tools publish share a trigger, so a fresh host routinely races the publish of its own sha tag. Preflight re-runs every minute while a pull failure is the blocker, republishing through the existing Gate; spawns unblock without a restart. Repeat verdicts log only on change.
• lab --help now documents --container-image / --container-tools-image (gap found in the #220 diagnosis).

Docs: ADR-0054, ops.md (option table, agent-tools, container-runner), CONTEXT.md glossary.

Verified: go test (podmanx/config/cmd-lab; full suite via the package check), go vet, golangci-lint 0 issues, gofmt clean; nix build .#checks.x86_64-linux.nixos-module and .#checks.x86_64-linux.nixos-container-closure both pass (dirty tree exercises the CLI-version fallback; CI's clean checkout exercises the rev path). Not driven end-to-end in-session: this host has no podman yet (that's the bug) — first real exercise is the next deploy after merge.

Operator notes for dev-new (post-merge):

• The host gets podman + a lab:100000:65536 subid range on next deploy; /etc/subuid already has dominik:100000:65536, so set services.lab.container.subIdRange.start = 165536; in Cloonar/nixos (or accept the documented overlap).
• The agent-tools package on git.cloonar.com must be publicly readable — the new publish verify step will say so if not.
• First deploy after merge may briefly refuse container spawns while the publish job pushes the sha tags; the retry loop clears it automatically.

🤖 Generated with Claude Code

Closes #220 Diagnosing the dev-new rollout showed a pin bump alone can never activate the container runner: the enable switch lives in the host-config repo, and nothing tied an agent-tools image (whose baked-in labctl talks to this server) to the server's code rev. This PR makes the defaults carry the whole story, per ADR-0054 (new): **nix/module.nix** - `services.lab.container.enable` defaults **true** — `services.lab.enable` alone makes the host container-ready. `enable = false` is the explicit host-only opt-out and renders the byte-identical pre-container unit. - New `container.toolsImageRepo` (default `git.cloonar.com/cloonar/agent-tools`); `container.toolsImages` defaults to `<repo>:claude-<self.rev>` / `<repo>:codex-<self.rev>`, falling back to the versions.env CLI-version tags when the source has no rev (dirty tree), so local `nix flake check`/rebuilds keep working. **flake.nix checks** - Zero-diff guard moved to an explicit `enable = false` dummy (podman leak-check added); new asserts pin default-on provisioning: tools flag with both default refs, no `--container-image`, `Delegate=true`, `RuntimeDirectory=lab`, podman+passt on the unit PATH, default subid range with no container config at all; `virtualisation.podman` on by default; explicitly-emptied `toolsImages` still fails eval. **.forgejo/workflows/agent-tools.yml** - Publish leg runs on **every** push to main (path gate kept on the PR smoke leg only) and pushes `<provider>-<git sha>` tags alongside the CLI-version tags. - New verify step: logout, pull the sha tags into a scratch storage root, compare digests — anonymous pull is the deployment contract (the module provisions no registry credentials), so a package flipped private fails the publish loudly instead of failing every host's preflight quietly. **Go** - `podmanx.Result.HasPullFailure()` + a retry loop in cmd/lab: deploy and agent-tools publish share a trigger, so a fresh host routinely races the publish of its own sha tag. Preflight re-runs every minute while a pull failure is the blocker, republishing through the existing `Gate`; spawns unblock without a restart. Repeat verdicts log only on change. - `lab --help` now documents `--container-image` / `--container-tools-image` (gap found in the #220 diagnosis). **Docs**: ADR-0054, ops.md (option table, agent-tools, container-runner), CONTEXT.md glossary. **Verified**: `go test` (podmanx/config/cmd-lab; full suite via the package check), `go vet`, `golangci-lint` 0 issues, `gofmt` clean; `nix build .#checks.x86_64-linux.nixos-module` and `.#checks.x86_64-linux.nixos-container-closure` both pass (dirty tree exercises the CLI-version fallback; CI's clean checkout exercises the rev path). Not driven end-to-end in-session: this host has no podman yet (that's the bug) — first real exercise is the next deploy after merge. **Operator notes for dev-new** (post-merge): - The host gets podman + a `lab:100000:65536` subid range on next deploy; `/etc/subuid` already has `dominik:100000:65536`, so set `services.lab.container.subIdRange.start = 165536;` in Cloonar/nixos (or accept the documented overlap). - The agent-tools package on git.cloonar.com must be publicly readable — the new publish verify step will say so if not. - First deploy after merge may briefly refuse container spawns while the publish job pushes the sha tags; the retry loop clears it automatically. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dominik.polakovics added 1 commit 2026-07-24 00:49:05 +02:00

feat(nix,ci): container-ready by default — enable=true, rev-pinned agent-tools tags, verified pulls (ADR-0054) ... 19ef5cfe2c

The first real container-runner rollout (issue #220) proved the explicit
opt-in was a deploy-time trap: the pin bump went green and container
spawns stayed refused, because the switch lives in a host-config repo no
coding-lab merge can flip. Defaults are the only channel this repo
controls end to end:

- nix: services.lab.container.enable defaults TRUE — enabling lab means a
  container-ready host. toolsImages defaults to
  <toolsImageRepo>:{claude,codex}-<self.rev> (new toolsImageRepo option,
  default git.cloonar.com/cloonar/agent-tools), falling back to the
  versions.env CLI-version tags on a revless/dirty source. The zero-diff
  guarantee moves to the opt-OUT: enable=false renders the byte-identical
  pre-container unit, pinned by the reworked nixos-module checks.
- ci: the agent-tools publish leg runs on EVERY push to main (labctl is
  built from the repo inside the image, so every merge changes what the
  images should carry) and pushes <provider>-<git sha> tags alongside the
  CLI-version tags; a new step verifies the sha tags pull back
  anonymously into a scratch store with matching digests — anonymous pull
  IS the deployment contract (the module provisions no registry creds).
- go: cmd/lab retries preflight while a tools-image-pull failure is the
  blocker (new Result.HasPullFailure), republishing through the existing
  Gate — a fresh deploy routinely races the publish job for its own sha
  tag, and spawns must unblock without a restart. Repeat verdicts log
  only when the failure set changes. lab --help now documents the
  container flags (gap found during the #220 diagnosis).
- docs: ADR-0054 (amends ADR-0051/0052), ops.md option table + agent-
  tools + container-runner sections, CONTEXT.md glossary. subid-collision
  caveat documented (100000 default overlaps first-user allocations;
  override subIdRange or opt out).

Closes #220

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics added 1 commit 2026-07-24 11:46:09 +02:00

refactor(nix,ci): build agent-tools only on change; pin advances only behind a published image ... dac946bfd7

Review of the first design (per-merge <provider>-<sha> tags) called it
out: rebuilding both images on every merge is the wrong cost model. The
settled shape (ADR-0054 rewritten):

- agent-tools.yml: publish leg re-path-gated to containers/** — images
  rebuild only when their inputs change. Per-merge sha tags dropped; the
  anonymous scratch-store pull verification stays, now against the
  CLI-version tags.
- nix: toolsImages defaults to the CLI-version tags read from
  versions.env at eval — the committed pin: the file that drives the
  build supplies the consumer refs, so the defaults move exactly when a
  publish does. No self.rev branch, no dirty-tree special case.
- deploy.yml: new wait step — checkout the pushed rev, derive the
  default refs from its versions.env, and poll the registry (skopeo,
  anonymous) until they resolve before bumping the Cloonar/nixos pin;
  on timeout the deploy fails and the pin stays. A merge that changes
  agent-tools serializes deploy behind its own publish; every other
  merge passes instantly.
- preflight: pull-FIRST per configured ref each boot (the default tags
  are moving refs — a labctl refresh via dispatch re-pushes the same
  tag, and exists-first would freeze hosts on their first-ever pull),
  falling back to a locally cached image with a Result.Warnings entry
  when the registry is unreachable. Only pull-failure-with-no-cache
  still refuses spawns, and that case keeps the cmd/lab retry loop.

Closes #220

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics commented 2026-07-24 11:46:21 +02:00

Author

Owner

Copy link

Redesigned per review feedback (commit dac946b) — the per-merge sha-tag scheme in the PR description is gone; ADR-0054 was rewritten to the settled shape:

• Build on change only: the publish leg is path-gated to containers/** again — no image builds on ordinary merges. The per-merge <provider>-<sha> tags are dropped.
• The committed pin is versions.env: toolsImages now defaults to <toolsImageRepo>:claude-<CLAUDE_CODE_VERSION> / :codex-<CODEX_VERSION> read from containers/agent-tools/versions.env at eval. Same file drives the build and the consumer refs, so the defaults move exactly when a publish does. (Also removes the dirty-tree special case the rev design needed.)
• The pin advances only behind a completed publish: deploy.yml gains a wait step — checkout the pushed rev, derive the default refs from its versions.env, poll the registry anonymously (skopeo via nix) until they resolve, 30-min deadline. On timeout the deploy fails and the Cloonar/nixos pin stays put. Merges that don't touch agent-tools pass this instantly.
• Preflight now pulls first, per boot, instead of image exists-first: the version tags are moving refs (a labctl refresh via workflow_dispatch re-pushes the same tag), and exists-first would freeze every host on the first digest it ever pulled. Registry unreachable + cached image = degrade to cache with a logged warning; only pull-failure-with-no-cache refuses spawns, and that case keeps the retry loop from the first commit.
• The anonymous scratch-store pull verification in the publish job stays (now against the CLI-version tags).

Verified again: go test/vet/lint/gofmt clean; nixos-module and nixos-container-closure checks both pass.

**Redesigned per review feedback** (commit dac946b) — the per-merge sha-tag scheme in the PR description is gone; ADR-0054 was rewritten to the settled shape: - **Build on change only**: the publish leg is path-gated to `containers/**` again — no image builds on ordinary merges. The per-merge `<provider>-<sha>` tags are dropped. - **The committed pin is `versions.env`**: `toolsImages` now defaults to `<toolsImageRepo>:claude-<CLAUDE_CODE_VERSION>` / `:codex-<CODEX_VERSION>` read from `containers/agent-tools/versions.env` at eval. Same file drives the build and the consumer refs, so the defaults move exactly when a publish does. (Also removes the dirty-tree special case the rev design needed.) - **The pin advances only behind a completed publish**: `deploy.yml` gains a wait step — checkout the pushed rev, derive the default refs from its `versions.env`, poll the registry anonymously (skopeo via nix) until they resolve, 30-min deadline. On timeout the deploy fails and the Cloonar/nixos pin stays put. Merges that don't touch agent-tools pass this instantly. - **Preflight now pulls first, per boot**, instead of `image exists`-first: the version tags are moving refs (a labctl refresh via `workflow_dispatch` re-pushes the same tag), and exists-first would freeze every host on the first digest it ever pulled. Registry unreachable + cached image = degrade to cache with a logged warning; only pull-failure-with-no-cache refuses spawns, and that case keeps the retry loop from the first commit. - The anonymous scratch-store pull verification in the publish job stays (now against the CLI-version tags). Verified again: go test/vet/lint/gofmt clean; `nixos-module` and `nixos-container-closure` checks both pass.

dominik.polakovics commented 2026-07-24 12:08:08 +02:00

Author

Owner

Copy link

[autoland] verdict: pass

This was generated by AI while landing a PR.

PASS — validated, awaiting free-text merge confirmation.

Signal relied on: Forgejo CI aggregate success (labctl pr checks --wait 221) — ci / native, agent-tools / smoke, and ci-nix / flake-check (21m, which runs the nixos-module default-on asserts and the container.enable=false zero-diff guard). agent-tools / publish correctly skipped (no containers/** diff). Did not re-run any of it.

Conventions: Closes #220 present and valid. Commits are Conventional Commits (feat/refactor).

Diff reviewed against #220's intent — sound:

• podmanx preflight pull-first reorder is correct; cached-fallback → warning, only no-cache pull failure fails; HasPullFailure()-keyed retry loop in cmd/lab is well-scoped and tested.
• flake.nix zero-diff guard migrated from default to explicit enable=false dummy; default-on asserts pin provisioning with no container config.
• versions.env-derived toolsImages default parses cleanly (VERSION prefix won't match the SHA256 line).
• agent-tools.yml verify step reads /tmp/digest.${p} written by the push --digestfile step; deploy.yml skopeo wait-for-refs gate is sound.

CONCERNS (non-blocking): PR title is not Conventional Commits and is stale vs the redesign — it says 'rev-pinned agent-tools release tags', but commit dac946b dropped the rev/sha-tag scheme for versions.env-pinned tags. If the server squash-merges on the PR title, mainline gets a non-conventional, misleading subject. Suggest retitling in the forge UI before merge (e.g. 'feat(nix,ci): container-ready by default — enable=true, versions.env-pinned agent-tools tags, verified pulls'). Not a blocker.

[autoland] verdict: pass > *This was generated by AI while landing a PR.* **PASS** — validated, awaiting free-text merge confirmation. **Signal relied on:** Forgejo CI aggregate `success` (labctl pr checks --wait 221) — `ci / native`, `agent-tools / smoke`, and `ci-nix / flake-check` (21m, which runs the nixos-module default-on asserts and the container.enable=false zero-diff guard). `agent-tools / publish` correctly skipped (no containers/** diff). Did not re-run any of it. **Conventions:** `Closes #220` present and valid. Commits are Conventional Commits (feat/refactor). **Diff reviewed against #220's intent** — sound: - podmanx preflight pull-first reorder is correct; cached-fallback → warning, only no-cache pull failure fails; HasPullFailure()-keyed retry loop in cmd/lab is well-scoped and tested. - flake.nix zero-diff guard migrated from default to explicit enable=false dummy; default-on asserts pin provisioning with no container config. - versions.env-derived toolsImages default parses cleanly (VERSION prefix won't match the SHA256 line). - agent-tools.yml verify step reads /tmp/digest.${p} written by the push --digestfile step; deploy.yml skopeo wait-for-refs gate is sound. **CONCERNS (non-blocking):** PR title is not Conventional Commits and is stale vs the redesign — it says 'rev-pinned agent-tools release tags', but commit dac946b dropped the rev/sha-tag scheme for versions.env-pinned tags. If the server squash-merges on the PR title, mainline gets a non-conventional, misleading subject. Suggest retitling in the forge UI before merge (e.g. 'feat(nix,ci): container-ready by default — enable=true, versions.env-pinned agent-tools tags, verified pulls'). Not a blocker.

dominik.polakovics merged commit 9ca6857f3d into main 2026-07-24 12:09:25 +02:00

dominik.polakovics referenced this pull request from a commit 2026-07-24 12:09:27 +02:00

Merge pull request 'Container-ready by default: container.enable=true, rev-pinned agent-tools release tags, verified pulls' (#221) from lab/20260723-2144 into main

dominik.polakovics referenced this pull request 2026-07-25 09:17:18 +02:00

deploy: agent-tools wait step discards skopeo errors, masking the real failure #224

dominik.polakovics referenced this pull request from a commit 2026-07-25 09:20:18 +02:00

fix(ci): surface skopeo's stderr in the deploy wait step (issue #224)

dominik.polakovics referenced this pull request 2026-07-25 09:20:30 +02:00

deploy: surface skopeo's stderr in the agent-tools wait step #225

dominik.polakovics referenced this pull request 2026-07-25 09:40:17 +02:00

deploy: wait step builds nixpkgs#skopeo with multiple outputs — $skopeo path is two lines, exec always fails #226

dominik.polakovics referenced this pull request 2026-07-25 09:43:32 +02:00

deploy: select skopeo's ^out output in the wait step #227

dominik.polakovics referenced this pull request from a commit 2026-07-25 09:43:33 +02:00

fix(ci): select skopeo's ^out output in the deploy wait step (issue #226)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is broken

  

chore

  

enhancement

New feature or improvement

  

needs-info

Waiting on reporter for more information

  

needs-triage

Maintainer needs to evaluate this issue

  

ready-for-agent

Fully specified, ready for an AFK agent

  

ready-for-human

Requires human implementation

  

wontfix

Will not be actioned

No labels

bug

chore

enhancement

needs-info

needs-triage

ready-for-agent

ready-for-human

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Cloonar/coding-lab!221

No description provided.