Cloonar/coding-lab

Fork 0

Codex: dynamic model catalog from `codex debug models` + per-model efforts seam #156

New issue

Closed

opened 2026-07-13 14:22:10 +02:00 by dominik.polakovics · 1 comment

dominik.polakovics commented

2026-07-13 14:22:10 +02:00

Owner

Problem

The codex model/effort catalogs are pinned from codex debug models on 0.133.0 (internal/provider/codex/codex.go). The deployed 0.144.1 binary has already drifted: two new models (gpt-5.6-terra, gpt-5.6-luna), a new ultra effort, per-model effort lists, and a visibility field. Users can't pick models the binary supports; worse, effort support now varies by model and codex does NOT clamp — an unsupported model+effort combo 400s at the API (live-verified: gpt-5.5 + ultra → 400 "Invalid value: 'max'").

Decision (grill 2026-07-13)

Codex gets true dynamic discovery; claude-code stays pinned (it has no machine-readable model listing — drift detection is a separate issue).

Probe

Run codex debug models once at provider construction (boot), synchronously, through the adapter's existing Runner seam, hard timeout ~10s. Cache for process lifetime — the binary is nix-pinned, so a new binary implies a restart. No TTL, no background refresh.
Decode into a minimal struct (slug, display_name, visibility, priority, supported_reasoning_levels, default_reasoning_level); ignore unknown fields. Output is ~170KB — cap read generously.
On any probe failure (binary missing, timeout, bad JSON, nonzero exit): fall back to the compiled-in pinned catalog (today's list stays in source as the documented fallback) and log one loud structured warning with the probe error. No status surface, no metric, no push.
Boot log records catalog source (probe vs fallback) and binary version.

Mapping — trust the binary fully

Filter visibility == "list" (replaces the hardcoded codex-auto-review slug filter).
Order by priority ascending; first entry = spawn default (on 0.144.1 the bare default becomes gpt-5.6-terra; stored repo/global defaults still layer on top and skip-layer semantics already handle catalog shrink — a stored default not in the catalog falls through, only explicit per-spawn values 400).
Label = display_name. Effort labels from a lab-side map with title-case fallback for unknown values (ultra → "Ultra").

Per-model efforts seam — enriched model options

Models() entries grow per-model fields: efforts list (binary order) + default effort (default_reasoning_level).
Efforts() stays as the union catalog — still used by the repo/global defaults pickers, where a model-independent stored effort default is fine because skip-layer validation falls through at spawn.
Strict spawn validation (explicit request) validates effort against the resolved model's effort list → 400 on unsupported combo.
Effort fallback rule change: with every layer unset, the fallback becomes the model's reported default effort (codex: medium, was first-entry low). Providers without a reported default (claude) keep the first-entry rule.
Claude adapter adapts to the enriched interface: same effort list for every model (claude clamps itself), no per-model default.
providertest fakes + conformance suite updated: per-model effort lists non-empty, each default present in its list, clone/aliasing checks extended.

Composer / API

Settings API ships models each carrying their efforts + default effort.
Composer: effort options depend on the selected model. When switching model invalidates the current effort selection, snap to the new model's default effort; valid selections are kept.
Repo/global defaults pickers keep the flat union list (unchanged UX).

Docs

New ADR amending ADR-0037's pinned-catalog decision (pinned list = fallback only).
internal/compat/codex/compat.md: re-pin instructions shrink accordingly; visibility-filter and per-model-effort couplings documented as pinned probe-schema fields.

Out of scope

Claude-code drift detection (separate issue).
Gemini adapter catalog (#126 owns its own pinning).

## Problem The codex model/effort catalogs are pinned from `codex debug models` on 0.133.0 (`internal/provider/codex/codex.go`). The deployed 0.144.1 binary has already drifted: two new models (`gpt-5.6-terra`, `gpt-5.6-luna`), a new `ultra` effort, **per-model** effort lists, and a `visibility` field. Users can't pick models the binary supports; worse, effort support now varies by model and codex does NOT clamp — an unsupported model+effort combo 400s at the API (live-verified: `gpt-5.5` + `ultra` → 400 "Invalid value: 'max'"). ## Decision (grill 2026-07-13) **Codex gets true dynamic discovery; claude-code stays pinned** (it has no machine-readable model listing — drift detection is a separate issue). ### Probe - Run `codex debug models` **once at provider construction** (boot), synchronously, through the adapter's existing Runner seam, hard timeout ~10s. Cache for process lifetime — the binary is nix-pinned, so a new binary implies a restart. No TTL, no background refresh. - Decode into a minimal struct (slug, display_name, visibility, priority, supported_reasoning_levels, default_reasoning_level); ignore unknown fields. Output is ~170KB — cap read generously. - **On any probe failure** (binary missing, timeout, bad JSON, nonzero exit): fall back to the compiled-in pinned catalog (today's list stays in source as the documented fallback) and log one loud structured warning with the probe error. No status surface, no metric, no push. - Boot log records catalog source (probe vs fallback) and binary version. ### Mapping — trust the binary fully - Filter `visibility == "list"` (replaces the hardcoded `codex-auto-review` slug filter). - Order by `priority` ascending; **first entry = spawn default** (on 0.144.1 the bare default becomes `gpt-5.6-terra`; stored repo/global defaults still layer on top and skip-layer semantics already handle catalog shrink — a stored default not in the catalog falls through, only explicit per-spawn values 400). - Label = `display_name`. Effort labels from a lab-side map with title-case fallback for unknown values (`ultra` → "Ultra"). ### Per-model efforts seam — enriched model options - `Models()` entries grow per-model fields: efforts list (binary order) + default effort (`default_reasoning_level`). - `Efforts()` **stays as the union catalog** — still used by the repo/global defaults pickers, where a model-independent stored effort default is fine because skip-layer validation falls through at spawn. - Strict spawn validation (explicit request) validates effort against the **resolved model's** effort list → 400 on unsupported combo. - **Effort fallback rule change**: with every layer unset, the fallback becomes the model's reported default effort (codex: `medium`, was first-entry `low`). Providers without a reported default (claude) keep the first-entry rule. - Claude adapter adapts to the enriched interface: same effort list for every model (claude clamps itself), no per-model default. - providertest fakes + conformance suite updated: per-model effort lists non-empty, each default present in its list, clone/aliasing checks extended. ### Composer / API - Settings API ships models each carrying their efforts + default effort. - Composer: effort options depend on the selected model. When switching model invalidates the current effort selection, **snap to the new model's default effort**; valid selections are kept. - Repo/global defaults pickers keep the flat union list (unchanged UX). ### Docs - New ADR amending ADR-0037's pinned-catalog decision (pinned list = fallback only). - `internal/compat/codex/compat.md`: re-pin instructions shrink accordingly; visibility-filter and per-model-effort couplings documented as pinned probe-schema fields. ## Out of scope - Claude-code drift detection (separate issue). - Gemini adapter catalog (#126 owns its own pinning).

dominik.polakovics added the

ready-for-agent

label

2026-07-13 14:22:10 +02:00

dominik.polakovics referenced this issue

2026-07-13 14:22:33 +02:00

Claude Code: model-catalog drift detection via ~/.claude.json caches (spike) #157

dominik.polakovics referenced this issue from a commit

2026-07-13 15:13:03 +02:00

feat: codex dynamic model catalog from `codex debug models` + per-model efforts seam

dominik.polakovics referenced this issue from a pull request that will close it,

2026-07-13 15:13:30 +02:00

feat: codex dynamic model catalog from codex debug models + per-model efforts seam #158

dominik.polakovics commented

2026-07-13 18:52:37 +02:00

Author

Owner

This was generated by AI while landing a PR.

Landing audit — PR #158

Verdict: PASS (no blockers; four non-blocking nits below).

Verification signal relied on

Forgejo CI ci / native — green (success in 5m13s). Not re-run. That job is the full gate: SPA lint/format/test/build, go build + go test under the ui tag, and golangci-lint.
The separate ci-nix flake check is path-gated on **/*.nix, flake.lock, go.mod, go.sum — this diff touches none of them, so its absence is correct rather than a coverage gap.
Mergeable against main (branch is behind by the #153 merge; merge-tree resolves clean, no conflicts).

Checked

Conventions: title is Conventional Commits; body carries a working Closes #156; head afk/156 authored as dominik.
Probe (internal/provider/codex/catalog.go): one-shot codex debug models at construction under a 10s budget; 8 MiB stdout cap kills a haywire binary rather than draining it; every malformed-catalog shape the conformance suite would flag (zero listed models, empty slug, empty/duplicate effort, duplicate slug) is a probe error, so the fallback is never bypassed by a bad catalog. New never fails on a probe error — one loud Warn, then the compiled-in 0.133.0 list.
Mapping: visibility=="list" filter, stable priority-ascending sort, display_name label with slug fallback, default_reasoning_level sanitized to "" when not a member of the model's own list.
Spawn resolution (internal/instance/credential.go): the effort pass now runs against the resolved model's list; explicit effort stays strict (→ the unsupported-combo 400), stored defaults skip-layer, and the all-unset fallback is the model's reported default before the first-entry rule. Model-less providers keep pre-#156 union semantics.
Aliasing: CloneModelOptions deep-copies the nested per-model effort lists (codex's and claude's catalog entries share one backing array, so a shallow clone would have let one caller poison every model). Conformance now pins this, plus per-model list well-formedness, default membership, and union coverage.
Composer: resolveEffortOption's candidate order (repo default → global default → default_effort → first entry) matches the server's layerSpawnDefault chain exactly. The composer always POSTs an explicit model+effort, and the server is strict on explicit efforts — so any client/server divergence would surface as a hard 400, not a silent wrong spawn. Stale-pick is impossible: the snap effect is untracked and runs after the provider-change reset.
Test ratio: 862 lines of test changes against ~1,000 lines of source.

Non-blocking nits (not gating the merge)

internal/httpapi/providers.go — Options gets a nil → [] guard but Models, Efforts, and the new nested ModelOption.Efforts do not, so a provider returning a nil slice would serialize null and throw in RunChat.tsx's header chip. Unreachable today (both adapters always return non-nil; conformance requires non-empty), but #156 adds a third nil-able slice. Relatedly, api.ts types efforts as non-nullable while spawn.ts defends with ?? [] — one of the two is wrong.
The server's model-less union fallback has no client mirror: a provider with an empty model catalog would spawn a union effort server-side while the composer hides the effort chip entirely. Unreachable with any conformant provider — really a comment/reality mismatch on the server side.
Test gaps: no web test composes an invalidating model switch with a stored effort default that stays valid on the new model (each half is tested, the composition is not); TestAPI_Providers never asserts the union efforts key is on the wire.
internal/provider/providertest/nolink.go — NoLinkFake.Models() returns the shared slice without CloneModelOptions, the exact aliasing the conformance check forbids. NoLinkFake never runs through Conformance(), so nothing fails today.

Behaviour change to expect on deploy

The deployed binary is 0.144.1, so the probe (not the fallback) will serve the catalog: the bare spawn default model becomes gpt-5.6-terra and the bare default effort becomes medium (was first-entry low). Both are the grill's explicit decisions, not drift.

> *This was generated by AI while landing a PR.* ## Landing audit — PR #158 **Verdict: PASS** (no blockers; four non-blocking nits below). ### Verification signal relied on - **Forgejo CI `ci / native` — green** (success in 5m13s). Not re-run. That job is the full gate: SPA lint/format/test/build, `go build` + `go test` under the `ui` tag, and golangci-lint. - The separate `ci-nix` flake check is path-gated on `**/*.nix`, `flake.lock`, `go.mod`, `go.sum` — this diff touches none of them, so its absence is correct rather than a coverage gap. - Mergeable against `main` (branch is behind by the #153 merge; merge-tree resolves clean, no conflicts). ### Checked - **Conventions**: title is Conventional Commits; body carries a working `Closes #156`; head `afk/156` authored as `dominik`. - **Probe** (`internal/provider/codex/catalog.go`): one-shot `codex debug models` at construction under a 10s budget; 8 MiB stdout cap kills a haywire binary rather than draining it; every malformed-catalog shape the conformance suite would flag (zero listed models, empty slug, empty/duplicate effort, duplicate slug) is a probe error, so the fallback is never bypassed by a bad catalog. `New` never fails on a probe error — one loud Warn, then the compiled-in 0.133.0 list. - **Mapping**: `visibility=="list"` filter, stable priority-ascending sort, `display_name` label with slug fallback, `default_reasoning_level` sanitized to `""` when not a member of the model's own list. - **Spawn resolution** (`internal/instance/credential.go`): the effort pass now runs against the **resolved model's** list; explicit effort stays strict (→ the unsupported-combo 400), stored defaults skip-layer, and the all-unset fallback is the model's reported default before the first-entry rule. Model-less providers keep pre-#156 union semantics. - **Aliasing**: `CloneModelOptions` deep-copies the nested per-model effort lists (codex's and claude's catalog entries share one backing array, so a shallow clone would have let one caller poison every model). Conformance now pins this, plus per-model list well-formedness, default membership, and union coverage. - **Composer**: `resolveEffortOption`'s candidate order (repo default → global default → `default_effort` → first entry) matches the server's `layerSpawnDefault` chain exactly. The composer always POSTs an explicit model+effort, and the server is strict on explicit efforts — so any client/server divergence would surface as a hard 400, not a silent wrong spawn. Stale-pick is impossible: the snap effect is untracked and runs after the provider-change reset. - **Test ratio**: 862 lines of test changes against ~1,000 lines of source. ### Non-blocking nits (not gating the merge) 1. `internal/httpapi/providers.go` — `Options` gets a `nil → []` guard but `Models`, `Efforts`, and the new nested `ModelOption.Efforts` do not, so a provider returning a nil slice would serialize `null` and throw in `RunChat.tsx`'s header chip. Unreachable today (both adapters always return non-nil; conformance requires non-empty), but #156 adds a third nil-able slice. Relatedly, `api.ts` types `efforts` as non-nullable while `spawn.ts` defends with `?? []` — one of the two is wrong. 2. The server's model-less union fallback has no client mirror: a provider with an empty model catalog would spawn a union effort server-side while the composer hides the effort chip entirely. Unreachable with any conformant provider — really a comment/reality mismatch on the server side. 3. Test gaps: no web test composes an *invalidating model switch* with a **stored** effort default that stays valid on the new model (each half is tested, the composition is not); `TestAPI_Providers` never asserts the union `efforts` key is on the wire. 4. `internal/provider/providertest/nolink.go` — `NoLinkFake.Models()` returns the shared slice without `CloneModelOptions`, the exact aliasing the conformance check forbids. `NoLinkFake` never runs through `Conformance()`, so nothing fails today. ### Behaviour change to expect on deploy The deployed binary is 0.144.1, so the probe (not the fallback) will serve the catalog: the bare spawn default model becomes `gpt-5.6-terra` and the bare default effort becomes `medium` (was first-entry `low`). Both are the grill's explicit decisions, not drift.

dominik.polakovics closed this issue

2026-07-13 19:22:53 +02:00

dominik.polakovics referenced this issue from a commit

2026-07-13 21:12:30 +02:00

fix(gemini): port Models() to the enriched ModelOption catalog (#156)

dominik.polakovics referenced this issue from a commit

2026-07-20 19:14:26 +02:00

feat(autoland): per-repo lander model/effort; fix runs resolve on the AFK chain

dominik.polakovics referenced this issue

2026-07-20 19:14:47 +02:00

feat(autoland): per-repo lander model/effort; fix runs resolve on the AFK chain #190

dominik.polakovics referenced this issue