feat(provider): two-tier adapter conformance bar — hermetic Tier-1 suite, live-spike Tier-2 checklist (ADR-0036) #88

Merged

dominik.polakovics merged 1 commit from afk/80 into main

2026-07-10 02:12:22 +02:00

dominik.polakovics commented

2026-07-10 02:00:00 +02:00

Owner

Makes "passes the conformance suite + checklist" the executable definition of done for any AgentProvider adapter, in the two enforced tiers decided in the 2026-07-09 provider-seam grill. Pins the rules landed by #75, #77, #78, and #79.

Tier 1 — hermetic conformance suite (CI, every adapter)

providertest.Conformance(t, p, fx) runs ten obligations against any adapter with no agent CLI, tmux, or network, every check through the product path:

patterns-dialect — ScrubPatterns/SeededPathPatterns compile via provider.CompileScrubPatterns (ADR-0033's boot gate, not a reimplementation).
scrub-markers — the adapter's real attribution samples (per-adapter Fixture vectors) must be stripped by its own patterns under both engines: RE2 and the pre-push hook's actual grep -i BRE — which is what makes the BRE∩RE2 common-dialect rule executable ((foo|bar) compiles as RE2 but means literal parens under BRE) — plus the Registry.ScrubRegexps union the agent-API body sanitizer runs. Optional clean samples guard against over-broad patterns.
seeding-exclude-coverage / seeding-incogni — the provider's own SeedWorkspace plus the real internal/seeder run in production order inside a hermetic linked worktree; empty git status --porcelain is the executable form of "ExcludeEntries cover every seeded path / no tracked-file writes", with anti-vacuous existence checks and per-file SeededPathPattern coverage.
context-file-lab-owned (ADR-0035 .local rule), spawn-argv (argv(prompt) == argv(no prompt) + [prompt]; declared spawn options must round-trip), catalogs, seedmeta-clone (mutation test), auth-flow (kind vocabulary), login-session (tmuxx.LoginSessionName/IsLoginSession, ADR-0034).

Adapters register in one table (internal/provider/conformance_test.go) — a future adapter gets full coverage by adding one entry. claude-code passes all ten in ordinary CI. The suite's own tests run deliberately-broken adapters (tracked AGENTS.md name, RE2-invalid pattern, prompt not trailing, uncovered seeded file, aliased slices, unknown auth kind, over-broad pattern) and assert each failure message names the obligation and its rule source.

Tier 2 — required live spikes (documented, not CI-run)

Held to the ADR-0008 bar and evidenced by a committed compat record at internal/compat/<providerID>/compat.md (claude-code's existing internal/compat/compat.md stays put as the worked example): transcript located by cwd + NEW identity on context clear, reply/dialog/interrupt recipes hazard-checked against the real TUI, context-file discovery verifiably wired, incogni attribution ground truth reconciled against ScrubPatterns.

Docs

docs/agents/provider-authoring.md — the ordered author path + the four no-escape-hatch hard requirements; issue #2 (Codex) adopts "Tier 1 green + Tier 2 evidenced" as its definition of done.
docs/adr/0036-two-tier-provider-conformance.md — records the bar.
CONTEXT.md — glossary entry.

Verification

go test ./... — all 31 packages green (includes TestConformance/claude-code, 13 suite self-tests, the neutrality guard, and untouched seeder/agentapi/compat suites).
gofmt -l clean, go vet ./... clean (golangci-lint runs in CI).

Closes #80

🤖 Generated with Claude Code

https://claude.ai/code/session_011ETaoGp2quNA465CTbUgW1

Makes "passes the conformance suite + checklist" the executable definition of done for any `AgentProvider` adapter, in the two enforced tiers decided in the 2026-07-09 provider-seam grill. Pins the rules landed by #75, #77, #78, and #79. ## Tier 1 — hermetic conformance suite (CI, every adapter) `providertest.Conformance(t, p, fx)` runs ten obligations against any adapter with no agent CLI, tmux, or network, every check through the product path: - **patterns-dialect** — ScrubPatterns/SeededPathPatterns compile via `provider.CompileScrubPatterns` (ADR-0033's boot gate, not a reimplementation). - **scrub-markers** — the adapter's real attribution samples (per-adapter `Fixture` vectors) must be stripped by its own patterns under **both** engines: RE2 and the pre-push hook's actual `grep -i` BRE — which is what makes the BRE∩RE2 common-dialect rule executable (`(foo|bar)` compiles as RE2 but means literal parens under BRE) — plus the `Registry.ScrubRegexps` union the agent-API body sanitizer runs. Optional clean samples guard against over-broad patterns. - **seeding-exclude-coverage / seeding-incogni** — the provider's own `SeedWorkspace` plus the real `internal/seeder` run in production order inside a hermetic linked worktree; empty `git status --porcelain` is the executable form of "ExcludeEntries cover every seeded path / no tracked-file writes", with anti-vacuous existence checks and per-file SeededPathPattern coverage. - **context-file-lab-owned** (ADR-0035 `.local` rule), **spawn-argv** (`argv(prompt) == argv(no prompt) + [prompt]`; declared spawn options must round-trip), **catalogs**, **seedmeta-clone** (mutation test), **auth-flow** (kind vocabulary), **login-session** (`tmuxx.LoginSessionName`/`IsLoginSession`, ADR-0034). Adapters register in **one table** (`internal/provider/conformance_test.go`) — a future adapter gets full coverage by adding one entry. claude-code passes all ten in ordinary CI. The suite's own tests run deliberately-broken adapters (tracked `AGENTS.md` name, RE2-invalid pattern, prompt not trailing, uncovered seeded file, aliased slices, unknown auth kind, over-broad pattern) and assert each failure message names the obligation and its rule source. ## Tier 2 — required live spikes (documented, not CI-run) Held to the ADR-0008 bar and evidenced by a committed compat record at `internal/compat/<providerID>/compat.md` (claude-code's existing `internal/compat/compat.md` stays put as the worked example): transcript located by cwd + NEW identity on context clear, reply/dialog/interrupt recipes hazard-checked against the real TUI, context-file discovery verifiably wired, incogni attribution ground truth reconciled against ScrubPatterns. ## Docs - `docs/agents/provider-authoring.md` — the ordered author path + the four no-escape-hatch hard requirements; issue #2 (Codex) adopts "Tier 1 green + Tier 2 evidenced" as its definition of done. - `docs/adr/0036-two-tier-provider-conformance.md` — records the bar. - `CONTEXT.md` — glossary entry. ## Verification - `go test ./...` — all 31 packages green (includes `TestConformance/claude-code`, 13 suite self-tests, the neutrality guard, and untouched seeder/agentapi/compat suites). - `gofmt -l` clean, `go vet ./...` clean (golangci-lint runs in CI). Closes #80 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_011ETaoGp2quNA465CTbUgW1

dominik.polakovics added 1 commit

2026-07-10 02:00:01 +02:00

feat(provider): two-tier adapter conformance bar — hermetic Tier-1 suite, live-spike Tier-2 checklist (ADR-0036)

ci / native (pull_request) Successful in 5m26s

Details

ea60df0474

The AgentProvider seam contract lived in interface doc comments, ADRs
(0008/0016/0017/0026/0030/0033/0034/0035), and claude-code as the worked
example — nothing executable stopped a second adapter (#2 Codex, Gemini)
from silently missing an obligation: the BRE∩RE2 pattern dialect, the
trailing-positional prompt shape, the lab-owned .local context-file name,
exclude coverage of seeded paths.

Tier 1: providertest.Conformance(t, p, fx) runs ten obligations against
any adapter, hermetically (no agent CLI, tmux, or network). Every check
goes through the product path, never a reimplementation: patterns compile
via CompileScrubPatterns; the fixture's real attribution samples must be
stripped by the adapter's own patterns under BOTH engines — RE2 and the
pre-push hook's actual `grep -i` BRE (what makes the common-dialect rule
executable: `(foo|bar)` compiles as RE2 but means literal parens under
BRE) — plus the Registry.ScrubRegexps union the body sanitizer runs. The
seeding checks run the provider's own SeedWorkspace plus the real
internal/seeder in production order inside a hermetic linked worktree;
an empty `git status --porcelain` is the executable form of
"ExcludeEntries cover every seeded path" (incogni variant included), with
anti-vacuous stats on the context file and skills dir and per-file
SeededPathPattern coverage. Argv shape is pinned by one equality —
argv(prompt) == argv(no prompt) + [prompt] — and every declared spawn
option must round-trip (explicit default == unset; negated default is
observable). Catalogs, AuthFlow kind vocabulary, defensive clones, and
the #77 login-session derivation round it out.

Adapters register in ONE table (internal/provider/conformance_test.go):
adding an adapter to lab means adding one entry constructing it into
t.TempDir() with its ground-truth marker fixture. claude-code passes all
ten in ordinary CI. Check functions return their violations as errors and
Conformance is only the *testing.T front-end, so the suite's own tests
run deliberately-broken adapters (tracked AGENTS.md name, RE2-invalid
pattern, prompt not trailing, uncovered seeded file, aliased slices,
unknown auth kind, over-broad scrub pattern) and assert the failure
message names the obligation and the rule's source.

Tier 2 stays out of CI (the ADR-0008 line: no real CLI or credentials
there) but becomes a named, evidenced merge gate: four live spikes —
transcript identity across a context clear, reply/dialog/interrupt
recipes hazard-checked against the real TUI, context-file discovery
(the CLI verifiably reads the lab-owned name), incogni attribution
ground truth — recorded at internal/compat/<providerID>/compat.md,
claude-code's existing internal/compat/compat.md as the worked example.

docs/agents/provider-authoring.md is the ordered author path with the
four no-escape-hatch hard requirements; "Tier 1 green + Tier 2 evidenced"
is the definition of done #2 adopts. ADR-0036 records the bar; CONTEXT.md
gains the glossary entry.

Closes #80

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011ETaoGp2quNA465CTbUgW1