Cloonar/coding-lab
3
0
Fork 0
Code Issues 23 Pull requests 4 Projects Releases Packages 1 Wiki Activity Actions

Provider conformance suite + provider-author checklist: executable seam obligations, live-spike merge gate #80

New issue
Closed
opened 2026-07-09 21:46:42 +02:00 by dominik.polakovics · 2 comments
dominik.polakovics commented 2026-07-09 21:46:42 +02:00
Owner
Copy link

Context

The AgentProvider seam contract lives in interface comments (internal/provider/provider.go), ADRs (0008, 0016, 0017, 0026, 0030), and claude-code's example. Nothing executable stops a new adapter (#2 Codex, Gemini) from silently missing an obligation — the clear/epoch rotation rule, the argv prompt-trailing shape, the SeedMeta pattern dialect. Decided in provider-seam grill (2026-07-09): 'passes the conformance suite + checklist' becomes the definition of done for any adapter, in two enforced tiers.

Design

Tier 1 — hermetic conformance suite (internal/provider/providertest, e.g. providertest.Conformance(t, provider)), run in CI for every registered adapter:

  • SeedMeta rules: ScrubPatterns/SeededPathPatterns compile in the BRE∩RE2 common dialect (#75's contract); ContextFileName follows the lab-owned .local-suffix rule (#79); ExcludeEntries cover every seeded path; declared slices are defensively cloned (mutation test).
  • SpawnArgv shape: initial prompt rides as the trailing positional; empty prompt appends nothing; declared spawn options are applied by the provider (fake options round-trip).
  • Catalogs non-empty and self-consistent (default model/effort are members).
  • AuthFlow.Kind is a recognized constant.
  • Login-session name derives per-provider and satisfies tmuxx.IsLoginSession (#77).
  • Incogni: SeedWorkspace(Incogni: true) produces a worktree whose seeded files carry no tracked-file writes, and the provider's ScrubPatterns strip its OWN attribution samples through the #75 sanitizer mechanism (each adapter ships marker samples as test vectors).

Tier 2 — required live spikes (ADR-0008 bar; run on a host with the real CLI, tagged like live_recipes_test.go), required per adapter and evidenced by a committed compat.md:

  • Transcript: locate by cwd, read/map to the universal schema, NEW identity on context clear (rotate or synthesized epoch).
  • Reply / dialog-answer / interrupt keystroke recipes hazard-checked against the real TUI (double-Esc/Ctrl-C must never kill the session).
  • Context-file discovery wiring: the CLI verifiably reads the lab-owned context file (e.g. Codex ↔ AGENTS.local.md via project config).
  • Incogni attribution ground truth: what the CLI actually writes into commits/PRs, with the adapter's ScrubPatterns reconciled against it.

Checklist (docs/agents/provider-authoring.md): the ordered author path — implement seam, declare SeedMeta, register config map entries (#78), pass Tier 1, run Tier 2 spikes, commit compat.md, grep-clean core. States the HARD requirements: durable locatable transcript (verified available for Codex rollout JSONLs and Gemini CLI session recording — no capability escape hatch), lab-owned context-file name, live verification before merge.

Sequencing

Pins rules established by #75, #77, #78, #79 — blocked until those merge; re-triage to ready-for-agent then.

Acceptance

  • claude-code passes Tier 1 in CI; the suite runs automatically for every adapter registered in tests.
  • Checklist doc exists and #2 can adopt 'passes suite + checklist' as its definition of done.
  • A deliberately-broken fake adapter (bad pattern dialect, tracked context-file name, prompt not trailing) fails the suite with actionable messages.
  • ADR recording the two-tier bar.
## Context The AgentProvider seam contract lives in interface comments (`internal/provider/provider.go`), ADRs (0008, 0016, 0017, 0026, 0030), and claude-code's example. Nothing executable stops a new adapter (#2 Codex, Gemini) from silently missing an obligation — the clear/epoch rotation rule, the argv prompt-trailing shape, the SeedMeta pattern dialect. Decided in provider-seam grill (2026-07-09): 'passes the conformance suite + checklist' becomes the definition of done for any adapter, in two enforced tiers. ## Design **Tier 1 — hermetic conformance suite** (`internal/provider/providertest`, e.g. `providertest.Conformance(t, provider)`), run in CI for every registered adapter: - `SeedMeta` rules: ScrubPatterns/SeededPathPatterns compile in the BRE∩RE2 common dialect (#75's contract); `ContextFileName` follows the lab-owned `.local`-suffix rule (#79); `ExcludeEntries` cover every seeded path; declared slices are defensively cloned (mutation test). - `SpawnArgv` shape: initial prompt rides as the trailing positional; empty prompt appends nothing; declared spawn options are applied by the provider (fake options round-trip). - Catalogs non-empty and self-consistent (default model/effort are members). - `AuthFlow.Kind` is a recognized constant. - Login-session name derives per-provider and satisfies `tmuxx.IsLoginSession` (#77). - Incogni: `SeedWorkspace(Incogni: true)` produces a worktree whose seeded files carry no tracked-file writes, and the provider's ScrubPatterns strip its OWN attribution samples through the #75 sanitizer mechanism (each adapter ships marker samples as test vectors). **Tier 2 — required live spikes** (ADR-0008 bar; run on a host with the real CLI, tagged like `live_recipes_test.go`), required per adapter and evidenced by a committed `compat.md`: - Transcript: locate by cwd, read/map to the universal schema, NEW identity on context clear (rotate or synthesized epoch). - Reply / dialog-answer / interrupt keystroke recipes hazard-checked against the real TUI (double-Esc/Ctrl-C must never kill the session). - Context-file discovery wiring: the CLI verifiably reads the lab-owned context file (e.g. Codex ↔ `AGENTS.local.md` via project config). - Incogni attribution ground truth: what the CLI actually writes into commits/PRs, with the adapter's ScrubPatterns reconciled against it. **Checklist** (`docs/agents/provider-authoring.md`): the ordered author path — implement seam, declare SeedMeta, register config map entries (#78), pass Tier 1, run Tier 2 spikes, commit compat.md, grep-clean core. States the HARD requirements: durable locatable transcript (verified available for Codex rollout JSONLs and Gemini CLI session recording — no capability escape hatch), lab-owned context-file name, live verification before merge. ## Sequencing Pins rules established by #75, #77, #78, #79 — **blocked until those merge**; re-triage to `ready-for-agent` then. ## Acceptance - claude-code passes Tier 1 in CI; the suite runs automatically for every adapter registered in tests. - Checklist doc exists and #2 can adopt 'passes suite + checklist' as its definition of done. - A deliberately-broken fake adapter (bad pattern dialect, tracked context-file name, prompt not trailing) fails the suite with actionable messages. - ADR recording the two-tier bar.
dominik.polakovics commented 2026-07-10 01:30:47 +02:00
Author
Owner
Copy link

This was generated by AI during triage.

Agent Brief

Category: enhancement
Summary: Make "passes the conformance suite + checklist" the executable definition of done for any AgentProvider adapter, in two enforced tiers: a hermetic CI suite and a live-spike checklist evidenced by a committed compat record.

All four sequencing blockers named in the issue body have merged: #75 (provider-declared attribution markers, ADR-0033), #77 (per-provider login sessions, ADR-0034), #78 (generic per-provider host config, ADR-0034), #79 (lab-owned context-file rule). Their rules are now stable and this issue pins them.

Current behavior:
The AgentProvider seam contract lives only in interface documentation comments and ADRs (0008, 0016, 0017, 0026, 0030, 0033, 0034), plus the claude-code adapter as a worked example. The providertest package (next to the provider package) currently contains only consumer-side test doubles — a scriptable Fake and a NoLinkFake — and no conformance entry point. Nothing executable stops a new adapter (Codex #2, Gemini) from silently missing an obligation: the pattern-dialect rule for scrub/seeded-path patterns, the trailing-positional prompt shape, the .local context-file naming rule, exclude-entry coverage of seeded files.

Desired behavior:

Tier 1 — hermetic conformance suite. A single entry point in the providertest package, shaped like Conformance(t *testing.T, p provider.AgentProvider) (possibly with a small per-adapter fixture struct for things like attribution marker samples), runnable in ordinary CI with no real agent CLI present. It must assert, per adapter:

  • SeedMeta().ScrubPatterns and SeedMeta().SeededPathPatterns compile in the BRE∩RE2 common dialect — via the same compilation path the product uses (CompileScrubPatterns and the pre-push guard's dialect), not a reimplementation.
  • SeedMeta().ContextFileName follows the lab-owned .local-suffix rule from #79 (a name the CLI's upstream tooling will never track or generate itself).
  • SeedMeta().ExcludeEntries cover every path the provider actually seeds (every seeded file must be matched by some exclude entry).
  • Slices returned from SeedMeta() are defensively cloned: mutating a returned slice and calling again yields the original values.
  • SpawnArgv(spec): the initial prompt rides as the trailing positional argument; an empty prompt appends nothing; declared spawn options (from the provider's Options() catalog) round-trip into argv when set on the spec.
  • Model/effort catalogs are non-empty and self-consistent: the declared defaults are members of their catalogs.
  • AuthFlow().Kind is one of the recognized AuthFlow* constants.
  • The provider's derived login-session name satisfies tmuxx.IsLoginSession (the #77 prefix rule).
  • Incogni: seeding a workspace with the incogni seed option writes no tracked files, and the provider's own ScrubPatterns strip that provider's attribution marker samples when run through the #75 sanitizer mechanism (each adapter supplies its real marker strings as test vectors; the union path Registry.UnionScrubPatterns must also strip them).

The suite runs in CI for every adapter registered in tests — claude-code today — so a new adapter gets conformance coverage by being added to one table, not by writing new assertions.

Tier 2 — required live spikes. Documented, not CI-run: the ADR-0008 bar. Per adapter, run on a host with the real CLI (in the style of the existing tagged live recipe tests) and evidenced by a committed per-adapter compat record (compat.md convention):

  • Transcript: locate by cwd, read and map to the universal schema, and confirm a NEW transcript identity on context clear (native rotation or synthesized epoch).
  • Reply / dialog-answer / interrupt keystroke recipes hazard-checked against the real TUI — double-Esc/Ctrl-C must never kill the session.
  • Context-file discovery: the CLI verifiably reads the lab-owned context file (e.g. Codex ↔ AGENTS.local.md via project config).
  • Incogni attribution ground truth: what the CLI actually writes into commits/PRs, reconciled against the adapter's ScrubPatterns.

Checklist doc. An agent-facing provider-authoring doc (alongside the existing docs/agents/ guides) giving the ordered author path: implement the seam → declare SeedMeta → register per-provider config map entries (#78) → pass Tier 1 → run Tier 2 spikes → commit the compat record → verify core stays adapter-agnostic (grep-clean of provider names outside the adapter package). It states the HARD requirements with no capability escape hatch: durable locatable transcript (verified available for Codex rollout JSONLs and Gemini CLI session recording), lab-owned context-file name, live verification before merge.

ADR. A new ADR records the two-tier bar and that "suite + checklist" is the definition of done for adapters.

Key interfaces:

  • provider.AgentProvider, provider.SeedMeta, provider.SpawnSpec, provider.AuthFlow and the AuthFlow* kind constants — the surfaces under test; the suite must not need any real CLI binary.
  • CompileScrubPatterns / Registry.UnionScrubPatterns (ADR-0033) — the real sanitizer path the incogni assertions must go through.
  • tmuxx.IsLoginSession (ADR-0034) — login-session name validation.
  • The existing providertest.Fake — stays as-is for consumer tests; the deliberately-broken adapters used to prove the suite fails are new, private to the suite's own tests.

Acceptance criteria:

  • claude-code passes Tier 1 in ordinary hermetic CI, via a per-adapter registration table such that future adapters are added with one entry.
  • A deliberately-broken fake adapter — bad pattern dialect, tracked (non-.local) context-file name, prompt not trailing, exclude entries missing a seeded path — fails the suite with actionable, obligation-naming messages (asserted in the suite's own tests).
  • The provider-authoring checklist doc exists and #2 (Codex) can adopt "passes suite + checklist" as its definition of done.
  • Tier 2 is specified in the checklist with the compat-record convention; claude-code's existing live evidence is linked or backfilled as the worked example.
  • An ADR records the two-tier conformance bar.

Out of scope:

  • Implementing any new adapter (Codex is #2; Gemini has no issue yet).
  • Changing the AgentProvider seam itself — the suite pins existing obligations; if an obligation turns out untestable as specified, flag it on the issue rather than reshaping the seam.
  • CI infrastructure changes (nix-store warming is #33).
  • Running Tier 2 spikes for not-yet-existing adapters.
> *This was generated by AI during triage.* ## Agent Brief **Category:** enhancement **Summary:** Make "passes the conformance suite + checklist" the executable definition of done for any `AgentProvider` adapter, in two enforced tiers: a hermetic CI suite and a live-spike checklist evidenced by a committed compat record. All four sequencing blockers named in the issue body have merged: #75 (provider-declared attribution markers, ADR-0033), #77 (per-provider login sessions, ADR-0034), #78 (generic per-provider host config, ADR-0034), #79 (lab-owned context-file rule). Their rules are now stable and this issue pins them. **Current behavior:** The `AgentProvider` seam contract lives only in interface documentation comments and ADRs (0008, 0016, 0017, 0026, 0030, 0033, 0034), plus the claude-code adapter as a worked example. The `providertest` package (next to the `provider` package) currently contains only consumer-side test doubles — a scriptable `Fake` and a `NoLinkFake` — and no conformance entry point. Nothing executable stops a new adapter (Codex #2, Gemini) from silently missing an obligation: the pattern-dialect rule for scrub/seeded-path patterns, the trailing-positional prompt shape, the `.local` context-file naming rule, exclude-entry coverage of seeded files. **Desired behavior:** *Tier 1 — hermetic conformance suite.* A single entry point in the `providertest` package, shaped like `Conformance(t *testing.T, p provider.AgentProvider)` (possibly with a small per-adapter fixture struct for things like attribution marker samples), runnable in ordinary CI with no real agent CLI present. It must assert, per adapter: - `SeedMeta().ScrubPatterns` and `SeedMeta().SeededPathPatterns` compile in the BRE∩RE2 common dialect — via the same compilation path the product uses (`CompileScrubPatterns` and the pre-push guard's dialect), not a reimplementation. - `SeedMeta().ContextFileName` follows the lab-owned `.local`-suffix rule from #79 (a name the CLI's upstream tooling will never track or generate itself). - `SeedMeta().ExcludeEntries` cover every path the provider actually seeds (every seeded file must be matched by some exclude entry). - Slices returned from `SeedMeta()` are defensively cloned: mutating a returned slice and calling again yields the original values. - `SpawnArgv(spec)`: the initial prompt rides as the trailing positional argument; an empty prompt appends nothing; declared spawn options (from the provider's `Options()` catalog) round-trip into argv when set on the spec. - Model/effort catalogs are non-empty and self-consistent: the declared defaults are members of their catalogs. - `AuthFlow().Kind` is one of the recognized `AuthFlow*` constants. - The provider's derived login-session name satisfies `tmuxx.IsLoginSession` (the #77 prefix rule). - Incogni: seeding a workspace with the incogni seed option writes no tracked files, and the provider's own `ScrubPatterns` strip that provider's attribution marker samples when run through the #75 sanitizer mechanism (each adapter supplies its real marker strings as test vectors; the union path `Registry.UnionScrubPatterns` must also strip them). The suite runs in CI for every adapter registered in tests — claude-code today — so a new adapter gets conformance coverage by being added to one table, not by writing new assertions. *Tier 2 — required live spikes.* Documented, not CI-run: the ADR-0008 bar. Per adapter, run on a host with the real CLI (in the style of the existing tagged live recipe tests) and evidenced by a committed per-adapter compat record (`compat.md` convention): - Transcript: locate by cwd, read and map to the universal schema, and confirm a NEW transcript identity on context clear (native rotation or synthesized epoch). - Reply / dialog-answer / interrupt keystroke recipes hazard-checked against the real TUI — double-Esc/Ctrl-C must never kill the session. - Context-file discovery: the CLI verifiably reads the lab-owned context file (e.g. Codex ↔ `AGENTS.local.md` via project config). - Incogni attribution ground truth: what the CLI actually writes into commits/PRs, reconciled against the adapter's `ScrubPatterns`. *Checklist doc.* An agent-facing provider-authoring doc (alongside the existing `docs/agents/` guides) giving the ordered author path: implement the seam → declare `SeedMeta` → register per-provider config map entries (#78) → pass Tier 1 → run Tier 2 spikes → commit the compat record → verify core stays adapter-agnostic (grep-clean of provider names outside the adapter package). It states the HARD requirements with no capability escape hatch: durable locatable transcript (verified available for Codex rollout JSONLs and Gemini CLI session recording), lab-owned context-file name, live verification before merge. *ADR.* A new ADR records the two-tier bar and that "suite + checklist" is the definition of done for adapters. **Key interfaces:** - `provider.AgentProvider`, `provider.SeedMeta`, `provider.SpawnSpec`, `provider.AuthFlow` and the `AuthFlow*` kind constants — the surfaces under test; the suite must not need any real CLI binary. - `CompileScrubPatterns` / `Registry.UnionScrubPatterns` (ADR-0033) — the real sanitizer path the incogni assertions must go through. - `tmuxx.IsLoginSession` (ADR-0034) — login-session name validation. - The existing `providertest.Fake` — stays as-is for consumer tests; the deliberately-broken adapters used to prove the suite fails are new, private to the suite's own tests. **Acceptance criteria:** - [ ] claude-code passes Tier 1 in ordinary hermetic CI, via a per-adapter registration table such that future adapters are added with one entry. - [ ] A deliberately-broken fake adapter — bad pattern dialect, tracked (non-`.local`) context-file name, prompt not trailing, exclude entries missing a seeded path — fails the suite with actionable, obligation-naming messages (asserted in the suite's own tests). - [ ] The provider-authoring checklist doc exists and #2 (Codex) can adopt "passes suite + checklist" as its definition of done. - [ ] Tier 2 is specified in the checklist with the compat-record convention; claude-code's existing live evidence is linked or backfilled as the worked example. - [ ] An ADR records the two-tier conformance bar. **Out of scope:** - Implementing any new adapter (Codex is #2; Gemini has no issue yet). - Changing the `AgentProvider` seam itself — the suite pins existing obligations; if an obligation turns out untestable as specified, flag it on the issue rather than reshaping the seam. - CI infrastructure changes (nix-store warming is #33). - Running Tier 2 spikes for not-yet-existing adapters.
dominik.polakovics commented 2026-07-10 02:10:38 +02:00
Author
Owner
Copy link

This was generated by AI while landing a PR.

Landing audit — PR #88 — Verdict: PASS

Verification signal: CI status was not readable from this session (no forge token / curl), so I ran the project's own checks against the merged tree. afk/80 is already up to date with main — no merge conflict.

Checks run (Go 1.26.4, CGO_ENABLED=0):

  • go test ./internal/provider/... — green, including TestConformance/claude-code passing all ten obligations (patterns-dialect, context-file-lab-owned, seedmeta-clone, catalogs, spawn-argv, auth-flow, login-session, seeding-exclude-coverage, seeding-incogni, scrub-markers) plus the suite's broken-adapter self-tests.
  • go test ./... (untagged) — 30/31 packages green. The single failure is internal/tmuxx (8 real-tmux D17 subprocess integration tests, "session exited immediately"), which fails identically on origin/main and is a sandbox-tmux limitation — not touched by this PR.
  • gofmt -l on the three changed .go files — clean.

Signal reach: CI's native job runs the ui-tagged suite; this diff touches only internal/provider/*.go + docs (no web/ui-tagged files), so the untagged run faithfully exercises the changed code.

Conventions: Conventional-Commits title (feat(provider): …); body carries a working Closes #80; head afk/80 matches the AFK contract.

Diff review: two-tier design matches this issue and the referenced ADRs. Conformance splits error-returning check functions from the *testing.T front-end (enabling the self-tests); scrub-markers runs the real RE2 compile path, the registry union, and the pre-push hook's actual grep -i BRE engine with correct exit-code handling; seeding checks run a real linked worktree + the real internal/seeder and assert git status --porcelain empty rather than a hand-maintained path list. No divergence from stated intent.

Awaiting explicit free-text merge confirmation before merging PR #88.

> *This was generated by AI while landing a PR.* ## Landing audit — PR #88 — Verdict: PASS **Verification signal:** CI status was not readable from this session (no forge token / `curl`), so I ran the project's own checks against the merged tree. `afk/80` is already up to date with `main` — no merge conflict. **Checks run** (Go 1.26.4, `CGO_ENABLED=0`): - `go test ./internal/provider/...` — green, including `TestConformance/claude-code` passing all ten obligations (patterns-dialect, context-file-lab-owned, seedmeta-clone, catalogs, spawn-argv, auth-flow, login-session, seeding-exclude-coverage, seeding-incogni, scrub-markers) plus the suite's broken-adapter self-tests. - `go test ./...` (untagged) — 30/31 packages green. The single failure is `internal/tmuxx` (8 real-tmux D17 subprocess integration tests, "session exited immediately"), which fails identically on `origin/main` and is a sandbox-tmux limitation — **not** touched by this PR. - `gofmt -l` on the three changed `.go` files — clean. Signal reach: CI's `native` job runs the `ui`-tagged suite; this diff touches only `internal/provider/*.go` + docs (no web/`ui`-tagged files), so the untagged run faithfully exercises the changed code. **Conventions:** Conventional-Commits title (`feat(provider): …`); body carries a working `Closes #80`; head `afk/80` matches the AFK contract. **Diff review:** two-tier design matches this issue and the referenced ADRs. `Conformance` splits error-returning check functions from the `*testing.T` front-end (enabling the self-tests); `scrub-markers` runs the real RE2 compile path, the registry union, and the pre-push hook's actual `grep -i` BRE engine with correct exit-code handling; seeding checks run a real linked worktree + the real `internal/seeder` and assert `git status --porcelain` empty rather than a hand-maintained path list. No divergence from stated intent. Awaiting explicit free-text merge confirmation before merging PR #88.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
afk/255
afk/251
afk/250
afk/249
afk/247
afk/243
afk/235
afk/195
afk/230
afk/222
afk/218
afk/207
afk/201
afk/198
afk/194
afk/191
afk/176
afk/189
afk/182
afk/185
lab/20260718-0240
afk/165
afk/163
afk/160
afk/126
afk/159
afk/156
afk/151
afk/154
afk/149
afk/145
afk/137
afk/142
afk/140
afk/134
afk/132
No results found.
Labels
Clear labels   
bug
Something is broken

  
chore

   
enhancement
New feature or improvement

  
needs-info
Waiting on reporter for more information

  
needs-triage
Maintainer needs to evaluate this issue

  
ready-for-agent
Fully specified, ready for an AFK agent

  
ready-for-human
Requires human implementation

  
wontfix
Will not be actioned

No labels
bug
chore
enhancement
needs-info
needs-triage
ready-for-agent
ready-for-human
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Cloonar/coding-lab#80
No description provided.