feat(containers): agent-tools OCI images + registry CI pipeline #210

Merged

dominik.polakovics merged 13 commits from afk/203 into main 2026-07-23 01:43:01 +02:00

Conversation 3 Commits 13 Files changed 11 +789

dominik.polakovics commented 2026-07-22 22:23:26 +02:00

Owner

Copy link

What

• containers/agent-tools/: versions.env — the new CLI version catalog (claude-code 2.1.198, codex 0.133.0, both artifact-sha256-pinned; mirrors the compat-record pins, which stay the authority), Containerfile.claude, Containerfile.codex, build.sh, smoke-test.sh; plus a root .containerignore.
• .forgejo/workflows/agent-tools.yml: PR leg = local podman build + injection smoke test (no push, no secrets); release leg (push:main path-gated + workflow_dispatch) = re-smoke, push digest-pinned tags to git.cloonar.com, emit @sha256: refs to the job summary. Fails loud with provisioning instructions when FORGE_REGISTRY_TOKEN is absent.
• Docs: docs/ops.md § Agent-tools images (contents + tagging scheme), ADR-0051, CONTEXT.md glossary entry Agent-tools image.

Mechanism — why the CLIs run on glibc AND musl bases without the base libc

• claude: the linux-x64-musl native build's only external dependency is the musl loader. We bundle Alpine's loader in the image and rewrite the binary's PT_INTERP to /opt/lab/lib/ld-musl-x86_64.so.1 (patchelf, NOT a loader wrapper — claude re-execs process.execPath as its embedded ripgrep, which a wrapper would break). Verified empirically on the exact pinned 2.1.198 binary: the patched binary answers --version and serves ripgrep 14.1.1 under env -i. This makes /opt/lab a hard mount destination (ADR-0051).
• codex: upstream x86_64 musl build is static-pie — runs as-is.
• labctl: pure Go, CGO_ENABLED=0, statically linked (verified locally).

Verification

No podman exists in the agent sandbox (as the triage brief anticipated), so this PR's agent-tools smoke job is the end-to-end proof: it podman-builds both images and injects them into stock debian:stable-slim and alpine, running claude --version / codex --version / labctl --help. Verified locally: full Go suite + golangci-lint green (test-env quirk: suite needs /bin on PATH + short TMPDIR in this sandbox — pre-existing, unrelated to this diff); workflow YAML parses; labctl static build replicated with the exact Containerfile command inside a .containerignore-simulated context; both artifact URLs live and sha256s match Anthropic's per-version manifest / the GitHub release digest. If the CI runner turns out unable to run podman, the smoke job's podman info diagnostics are the evidence — flag back to the issue rather than merging unverified.

Operator prerequisites (release leg only)

• Provision FORGE_REGISTRY_TOKEN (Forgejo access token, write:package on git.cloonar.com) as a repo CI secret; optional FORGE_REGISTRY_USER when the token isn't owned by the repository-owner account. PRs never see either.

Out of scope per the brief: the container runner (#205), dev-image selection (#207), a gemini image (seam left in versions.env until #126 lands).

Closes #203

## What - `containers/agent-tools/`: **versions.env** — the new CLI version catalog (claude-code **2.1.198**, codex **0.133.0**, both artifact-sha256-pinned; mirrors the compat-record pins, which stay the authority), **Containerfile.claude**, **Containerfile.codex**, **build.sh**, **smoke-test.sh**; plus a root `.containerignore`. - `.forgejo/workflows/agent-tools.yml`: PR leg = local podman build + injection smoke test (no push, no secrets); release leg (push:main path-gated + workflow_dispatch) = re-smoke, push digest-pinned tags to git.cloonar.com, emit `@sha256:` refs to the job summary. Fails loud with provisioning instructions when `FORGE_REGISTRY_TOKEN` is absent. - Docs: `docs/ops.md` § Agent-tools images (contents + tagging scheme), **ADR-0051**, CONTEXT.md glossary entry **Agent-tools image**. ## Mechanism — why the CLIs run on glibc AND musl bases without the base libc - **claude**: the `linux-x64-musl` native build's only external dependency is the musl loader. We bundle Alpine's loader in the image and rewrite the binary's PT_INTERP to `/opt/lab/lib/ld-musl-x86_64.so.1` (patchelf, NOT a loader wrapper — claude re-execs `process.execPath` as its embedded ripgrep, which a wrapper would break). Verified empirically on the exact pinned 2.1.198 binary: the patched binary answers `--version` and serves ripgrep 14.1.1 under `env -i`. This makes **/opt/lab a hard mount destination** (ADR-0051). - **codex**: upstream x86_64 musl build is static-pie — runs as-is. - **labctl**: pure Go, `CGO_ENABLED=0`, statically linked (verified locally). ## Verification No podman exists in the agent sandbox (as the triage brief anticipated), so **this PR's agent-tools smoke job is the end-to-end proof**: it podman-builds both images and injects them into stock `debian:stable-slim` and `alpine`, running `claude --version` / `codex --version` / `labctl --help`. Verified locally: full Go suite + golangci-lint green (test-env quirk: suite needs `/bin` on PATH + short TMPDIR in this sandbox — pre-existing, unrelated to this diff); workflow YAML parses; labctl static build replicated with the exact Containerfile command inside a `.containerignore`-simulated context; both artifact URLs live and sha256s match Anthropic's per-version manifest / the GitHub release digest. If the CI runner turns out unable to run podman, the smoke job's `podman info` diagnostics are the evidence — flag back to the issue rather than merging unverified. ## Operator prerequisites (release leg only) - Provision `FORGE_REGISTRY_TOKEN` (Forgejo access token, `write:package` on git.cloonar.com) as a repo CI secret; optional `FORGE_REGISTRY_USER` when the token isn't owned by the repository-owner account. PRs never see either. Out of scope per the brief: the container runner (#205), dev-image selection (#207), a gemini image (seam left in versions.env until #126 lands). Closes #203

dominik.polakovics added 1 commit 2026-07-22 22:23:26 +02:00

feat(containers): agent-tools OCI images + registry CI pipeline (#203) ... 37ad09179c

Per-provider agent-tools images (claude, codex) built FROM scratch: the
provider CLI as a self-contained musl bundle plus a static labctl,
injected read-only at /opt/lab into any dev container.
containers/agent-tools/versions.env is the new CLI version catalog
(mirroring the compat-record pins); .forgejo/workflows/agent-tools.yml
smoke-tests injection into debian:stable-slim and alpine on PRs, and on
the release path pushes digest-pinned tags to git.cloonar.com, failing
loud with provisioning instructions when FORGE_REGISTRY_TOKEN is absent.
claude runs on both libcs via its musl build + bundled alpine loader +
PT_INTERP rewrite (patchelf; a loader wrapper would break the embedded
ripgrep self-exec — verified empirically on the pinned 2.1.198 binary).
Documented in docs/ops.md § Agent-tools images, ADR-0051, and a
CONTEXT.md glossary entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics commented 2026-07-22 22:26:24 +02:00

Author

Owner

Copy link

[autoland] verdict: reject

Automated lander verdict: FAIL — CI check aggregate is failure, so the PR is not landable.

Verification signal relied on: labctl pr checks 210.

Failing row (blocker):

• agent-tools / smoke (pull_request) — failure, "Failing after 48s"
  (run 234 / job 0: /Cloonar/coding-lab/actions/runs/234/jobs/0)

Context of the other rows:

• agent-tools / publish (pull_request) — skipped (expected; the publish leg is push/workflow_dispatch-only).
• ci / native (pull_request) — still pending at verdict time; does not affect the outcome since a red row already dominates (precedence failure > pending > success).

Why this blocks: the agent-tools / smoke job is this PR's own designated end-to-end proof (per the PR body: "this PR's agent-tools smoke job is the end-to-end proof"). With it red, the build+injection path is unverified and the PR cannot land. The fix belongs to the author, not the lander — per the validation core a red check is a blocker and this is not a trivial (formatting/lint/conflict) fix.

I could not read the job logs to pinpoint the exact failing step — the lander has no Forgejo API/log credential (git.cloonar.com returns 307→login; the run token is not a forge token). Please inspect the job at the URL above. The likely first suspects, given a ~48s failure:

1. The Ensure podman step — this is the first workflow to require podman on these runners; if the runner cannot initialize podman (rootless/subuid-subgid, /dev/fuse, cgroups), podman info fails here. The PR body anticipates exactly this: "If the CI runner turns out unable to run podman, the smoke job's podman info diagnostics are the evidence — flag back to the issue rather than merging unverified." If that is the cause, it is a runner-provisioning matter (docs/ops.md § "CI runner prerequisites" + podman) rather than a code defect, and needs a human/infra decision — surface it on #203.
2. An early failure in containers/agent-tools/build.sh claude (base-image pull or CLI-artifact fetch).

Read the Ensure podman and claude: build + injection smoke test step output to distinguish these, fix, and hand back with labctl pr rerequest.

Conventions were otherwise satisfied: Conventional Commits title, Closes #203 present, head branch afk/203 matches, and the diff is purely additive (589 insertions, no edits to existing logic). The sole blocker is the red check.

[autoland] verdict: reject Automated lander verdict: **FAIL** — CI check aggregate is `failure`, so the PR is not landable. Verification signal relied on: `labctl pr checks 210`. Failing row (blocker): - `agent-tools / smoke (pull_request)` — **failure**, "Failing after 48s" (run 234 / job 0: /Cloonar/coding-lab/actions/runs/234/jobs/0) Context of the other rows: - `agent-tools / publish (pull_request)` — skipped (expected; the publish leg is `push`/`workflow_dispatch`-only). - `ci / native (pull_request)` — still `pending` at verdict time; does not affect the outcome since a red row already dominates (precedence failure > pending > success). Why this blocks: the `agent-tools / smoke` job is this PR's own designated end-to-end proof (per the PR body: "this PR's agent-tools smoke job is the end-to-end proof"). With it red, the build+injection path is unverified and the PR cannot land. The fix belongs to the author, not the lander — per the validation core a red check is a blocker and this is not a trivial (formatting/lint/conflict) fix. I could not read the job logs to pinpoint the exact failing step — the lander has no Forgejo API/log credential (git.cloonar.com returns 307→login; the run token is not a forge token). Please inspect the job at the URL above. The likely first suspects, given a ~48s failure: 1. The `Ensure podman` step — this is the first workflow to require podman on these runners; if the runner cannot initialize podman (rootless/subuid-subgid, /dev/fuse, cgroups), `podman info` fails here. The PR body anticipates exactly this: "If the CI runner turns out unable to run podman, the smoke job's `podman info` diagnostics are the evidence — flag back to the issue rather than merging unverified." If that is the cause, it is a runner-provisioning matter (docs/ops.md § "CI runner prerequisites" + podman) rather than a code defect, and needs a human/infra decision — surface it on #203. 2. An early failure in `containers/agent-tools/build.sh claude` (base-image pull or CLI-artifact fetch). Read the `Ensure podman` and `claude: build + injection smoke test` step output to distinguish these, fix, and hand back with `labctl pr rerequest`. Conventions were otherwise satisfied: Conventional Commits title, `Closes #203` present, head branch `afk/203` matches, and the diff is purely additive (589 insertions, no edits to existing logic). The sole blocker is the red check.

dominik.polakovics added 1 commit 2026-07-22 22:37:00 +02:00

fix(containers): configure podman for the nested Docker-backed CI runner ... 819fafda73

The agent-tools smoke job (run 234) died mounting the first build layer:
the job executes inside the runner's Docker container, where podman's
default overlay storage can mount neither natively (overlay-upon-overlay)
nor via fuse-overlayfs (no /dev/fuse — 'fuse: device not found').

Configure podman for nested operation before first use, in both legs:

- vfs storage driver (needs no mounts at all) via a CI-scoped
  /etc/containers/storage.conf, plus cgroupfs/file backends in
  containers.conf (the job container has no systemd/journald)
- BUILDAH_ISOLATION=chroot job env: RUN steps skip the OCI runtime's
  namespace/cgroup needs and inherit the job's own network namespace
- smoke-test.sh podman run gains --network=host --cgroups=disabled: the
  probe needs neither, and skipping both drops the netavark/cgroup
  requirements — with no change to what the injection test proves

docs/ops.md § CI runner prerequisites documents the nested-runner config.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics added 1 commit 2026-07-22 22:38:13 +02:00

fix(containers): spell out runroot/graphroot in the CI storage.conf ... cb48466490

A hand-written /etc/containers/storage.conf replaces ALL built-in
storage defaults, so listing only the driver left podman failing with
"runroot must be set" (run 236). Add the stock runroot/graphroot paths
alongside the vfs driver in both legs' config.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics added 1 commit 2026-07-22 22:45:33 +02:00

fix(containers): stop vfs storage from exhausting the runner's disk ... ee534c9094

Run 238 proved the whole build+injection path works in CI (download,
sha verify, sanity run, patchelf, scratch COPYs all green) and then hit
ENOSPC committing the final layer: under vfs every per-step layer commit
is a full copy of the stage filesystem.

- BUILDAH_LAYERS=false in both legs: one commit per build stage instead
  of one per step; per-run stores start empty, so step-layer caching
  bought nothing in CI anyway
- podman image prune -f between provider builds (dangling stage
  leftovers only — tagged images survive, so the publish leg still
  pushes what it built)
- df -h diagnostics in Ensure podman and after each build step

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics added 1 commit 2026-07-22 22:49:24 +02:00

fix(containers): pin crun as the CI runtime for --cgroups=disabled ... 6b19463165

Run 240 got the claude image fully built and tagged (the disk fix
held) and then died starting the first smoke container: Ubuntu's podman
defaults to runc, which refuses --cgroups=disabled ("requested OCI
runtime runc is not compatible with NoCgroups"). Install crun alongside
podman and pin runtime = crun in the CI containers.conf; crun supports
cgroup-less containers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics added 1 commit 2026-07-22 23:06:07 +02:00

fix(containers): add curl stall protection to the pinned CLI downloads ... 228df0ac5e

CI run 242 sat >15min inside the claude download RUN step: curl -fsSL
with no timeouts rides a dead connection forever. Abort transfers that
drop under 10KB/s for 30s, cap connect at 15s, and retry 3x — in both
Containerfiles.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics added 1 commit 2026-07-22 23:11:43 +02:00

fix(containers): rbind the job's /proc into the CI smoke containers ... 0313ba21c6

Run 244 proved the whole build path green and moved the failure into
the smoke run itself: the runner's job container carries locked
masked-path overmounts on /proc, so the kernel's fully-visible rule
denies crun's fresh procfs mount ("mount `proc` to `proc`: Operation
not permitted").

smoke-test.sh gains an SMOKE_EXTRA_RUN_ARGS seam (default empty — real
hosts stay pristine); CI sets it to --pid=host plus a recursive bind of
the job's /proc, which sidesteps the fresh-procfs mount while leaving
the mechanism under test (image mount, PATH, loader, exec) untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics added 1 commit 2026-07-22 23:28:00 +02:00

fix(containers): abort and retry crawling CLI downloads ... 16b227cedc

Run 246 sat 15min inside the claude download again: the 10KB/s abort
floor never triggers on a degraded-but-alive connection. The CDN path is
bimodal (multi-MB/s or a crawl), so abort transfers under 250KB/s for
30s and retry up to 6 times; --retry-all-errors also covers mid-transfer
resets, made safe by the sha256 verification that follows every fetch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics added 1 commit 2026-07-22 23:50:54 +02:00

fix(containers): fetch pinned CLI artifacts once, cache them across CI runs ... 4030e58098

Runs 242-248 show downloads.claude.ai throttling our egress IP after
repeated pulls of the same 150MB pinned binary — every CI run re-fetched
it inside the image build, and by run 248 all retry attempts crawled
below the abort floor. The official install script confirms there is no
alternate host, so the fix is to stop re-downloading:

- build.sh gains fetch_artifact: download into
  containers/agent-tools/artifacts/ (gitignored), sha256-verify against
  versions.env, and reuse any file that already matches the pin
- the Containerfiles COPY the artifact from the build context and
  RE-verify its sha256 in-stage, so the build stays self-verifying and
  a missing pin still fails; sanity runs and patchelf are unchanged;
  the codex stage drops its apk curl install entirely
- both workflow legs cache the artifacts dir via actions/cache keyed on
  versions.env (restore-keys let a version bump refetch only the bumped
  artifact), so CI touches the CDN once per version — the same cache
  backend the native gate's setup-go/setup-node already require

Verified locally: codex fetch + sha256 + cached-reuse pass end-to-end
(PODMAN=true stub); the claude fetch crawls from this sandbox too, which
shares the throttled egress — evidence for, not against, the cache.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics added 1 commit 2026-07-22 23:55:49 +02:00

fix(containers): private-pidns smoke containers + bank fetched artifacts ... 734b1c007f

Two findings from run 250 (whose claude image built green end-to-end):

- crun refuses --cgroups=disabled combined with --pid=host ("containers
  not creating Cgroups must create a private PID namespace"). Drop
  --pid=host: the /proc rbind alone is what dodges the denied fresh
  procfs mount, and the rbound /proc merely shows the job's pidns view,
  which the version/help probes never read.
- actions/cache only saves on job SUCCESS, so run 250's good claude
  fetch was discarded when the smoke step failed. Split into
  cache/restore + cache/save with if: always() and a run-id-suffixed
  key (entries are immutable), so every fetched artifact is banked no
  matter how the job ends; sha256 pinning in build.sh makes restored
  files trustless either way.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics added 1 commit 2026-07-23 00:17:10 +02:00

fix(containers): private UTS namespace for the CI smoke containers ... e146388d19

Run 252 banked the claude artifact cache (the CDN cost is now once per
version) and moved the smoke failure one syscall further: podman couples
the UTS namespace to --network=host, and with a host UTS namespace
crun's hostname setup dies with "sethostname: Operation not permitted".
Pass --uts=private in SMOKE_EXTRA_RUN_ARGS so the hostname is set inside
a fresh UTS namespace the job's userns owns.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics added 1 commit 2026-07-23 00:37:50 +02:00

fix(containers): host UTS namespace — the runner seccomp denies sethostname ... ee6e1d05da

Run 254 failed with "crun: sethostname: Operation not permitted" even
under an explicit --uts=private, where the kernel would allow the call —
so the denial comes from the runner's Docker seccomp profile, not from
namespace ownership. Switch SMOKE_EXTRA_RUN_ARGS to --uts=host: podman
then omits the hostname from the OCI spec and crun never calls
sethostname at all.

(Run 254 also banked the claude artifact cache on fj-runner-2; with
fj-runner-1 banked by run 252, both runner hosts now restore the pinned
CLI without touching the CDN.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dominik.polakovics commented 2026-07-23 00:46:14 +02:00

Author

Owner

Copy link

[autoland] verdict: fix-done

[autoland] verdict: fix-done

dominik.polakovics commented 2026-07-23 00:47:11 +02:00

Author

Owner

Copy link

[autoland] verdict: pass

[autoland] verdict: pass

dominik.polakovics added 1 commit 2026-07-23 01:22:35 +02:00

Merge remote-tracking branch 'origin/main' into afk/203 5ba5a4b1aa

dominik.polakovics merged commit c9008871da into main 2026-07-23 01:43:01 +02:00

dominik.polakovics referenced this pull request from a commit 2026-07-23 01:43:02 +02:00

Merge pull request 'feat(containers): agent-tools OCI images + registry CI pipeline' (#210) from afk/203 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is broken

  

chore

  

enhancement

New feature or improvement

  

needs-info

Waiting on reporter for more information

  

needs-triage

Maintainer needs to evaluate this issue

  

ready-for-agent

Fully specified, ready for an AFK agent

  

ready-for-human

Requires human implementation

  

wontfix

Will not be actioned

No labels

bug

chore

enhancement

needs-info

needs-triage

ready-for-agent

ready-for-human

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Cloonar/coding-lab!210

No description provided.