Sidecar versioning: machine-owned canonical sidecars, harness migrate, in-memory migration chain #302

New issue

Closed

opened 2026-10-15 12:26:59 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-10-15 12:26:59 +02:00

Owner

This was generated by AI during triage.

Summary

Sidecars become machine-owned, versioned files that the harness migrates forward: a harness migrate command rewrites every sidecar and its language files to the current version in one canonical form, harness check warns about files that need it, and the loaders apply the same migration in memory so an unmigrated site always opens. This is what lets the sidecar schema change after sites are in production without pinning the harness.

Current behavior

The harness is never pinned (AGENTS.md "Installing the harness"): every site runs the latest release. Nothing today lets a release change the shape of a sidecar, because every sidecar on every customer site would have to be rewritten by hand.
Every sidecar file carries version. Page types, backend layouts and editor modules pin it to exactly 1 through sidecardir.RequireVersion1; element sidecars (internal/element/load.go) accept any integer >= 1 and ignore it. Every shipped sidecar is version 1.
Sidecars are hand-formatted YAML with long explanatory comments (scaffold/editor/elements/*.yaml, the bundles). The sidecar drafter (internal/sidecar) writes a # NOTE comment in one case.
Language files (<name>.<code>.yaml, "Sidecar language files" in AGENTS.md) carry no version; they mirror the sidecar's keys and are read only after their sidecar loaded.
A feature receipt (editor/features/<name>.yaml, docs/harness.md "The receipt") records the SHA-256 of every vendored file as written by harness add. Nothing ever re-derives it; a file whose bytes differ counts as customised, which makes the feature "in use" and blocks harness remove behind --purge.
harness check already warns about a content file that is not canonical; there is no equivalent for sidecars.
deckle-hosted runs harness serve directly on a site's worktree and never runs harness check before opening the editor. It parses harness.yaml and feature.yaml, never sidecars.

Desired behavior

Versions. Each sidecar family has a current version the harness knows. A file at the current version loads as today. A file at an older supported version loads through the migration chain in memory, in every loader (harness serve, harness check, harness add, …), so no release ever locks a site out of its editor. A file above the current version, or below the oldest supported one, is a load error naming the file and the version.

Canonical form. The harness owns the sidecar's serialization the way the content engine owns the content format (ADR 0001): fixed key order, one spelling per value, no comments, and a header comment the serializer itself emits saying the file is machine-owned by the harness and that comments do not survive. Every sidecar the harness writes (harness init, harness add, harness sidecar, harness migrate) is canonical. The shipped scaffold and the embedded bundles are canonical and at the current version, enforced by a drift test.

harness migrate [--site <dir>] [--json]. Rewrites every sidecar of every family (elements, page types, backend layouts, editor modules) and each sidecar's language files to the current version in canonical form; a file already canonical and current is not touched. For every rewritten file listed in a feature receipt whose bytes matched the receipt's hash before the rewrite, the receipt's hash is updated; a file that did not match is left as customised. Exit 0 whether or not anything changed; exit 1 when a file cannot be migrated (it does not load even through the chain, or cannot be written), with every other file still migrated. Runs without Hugo, like harness add, so a host can run it in a clean clone. --json follows the add command's conventions: {"formatVersion": 1, "status": "migrated" | "unchanged", "files": [paths rewritten], "receipts": [receipt paths rewritten]}, and {"formatVersion": 1, "status": "error", "error": "…", "files": […]} on exit 1.

harness check. Warns, per file, about a sidecar or language file below the current version or not in canonical form, naming harness migrate as the fix. Both are warnings: the file still loads.

Loader and chain. One chain of per-version transforms per family, each a pure function from the raw document of version N and its language files to version N+1. The loader applies it in memory and harness migrate applies it to disk; nothing else reads an old version. Today the chain holds no transform (there is no version 2); the structure, its tests and the two call sites are the deliverable, with the canonicalization of a version 1 file as the first real rewrite.

Directory bundles. harness add --dir migrates a bundle's sidecars and language files while copying them, and the receipt records the bytes written.

Decisions

The harness stays unpinned and sidecars migrate forward — one harness version in production is the security stance; a pin would spread sites across releases.
The version is per file, not per site — a file stays self-describing, bundles vendor files, and a directory bundle may arrive at an older version.
Element sidecars are pinned to version 1 through sidecardir.RequireVersion1, the helper the other three families use — one owner of the rule, and a site must never hold a meaningless version 2 before version 2 has a meaning.
A bump means a shape change that needs a rewrite; a new optional key never bumps — the loader reads version N with or without the key, and a bump forces a commit on every customer repository.
Migrations change shape, never stored param names or value spellings — renaming a param is a content migration across every page of every site, a different and heavier thing (the category rename path is the precedent), and keeping the two apart is what makes sidecar migrations cheap enough to ship.
The transform is one function with two callers (loader in memory, migrate on disk) — "Every rule has exactly one owner" (CLAUDE.md); a loader that read old versions its own way would be a second copy.
Every version back to 1 stays supported; dropping one is a deliberate decision, never a side effect.
Sidecars are machine-owned and comments are dropped — the facts belong in AGENTS.md ("Every fact has one document"), and a comment-preserving migration would have to be a textual edit per version, the cost that keeps migrations rare. The header comment is the one comment, emitted by the serializer.
Language files carry no version; they are bound to their sidecar's — their keys mirror the sidecar's keys, so a transform that renames a structural key renames it in both, and the loader already reads a language file only after its sidecar loaded.
Receipt hashes are rewritten only for files whose bytes matched the receipt before migration — a customised file stays customised, which is what the hash exists to say; without the rewrite every bundle feature on every site would need --purge to remove after one migration, and the gap fill never overwrites a file, so the hash would never catch up.
Outdated and non-canonical are warnings, not errors — the file loads.
deckle-hosted runs harness migrate per site on each harness upgrade, commits to main and reconciles the editor branch, through the path a feature add already uses; that is a deckle-hosted issue. The --json shape here is the cross-repo contract it consumes, hence formatVersion.

Key interfaces

sidecardir.Load and sidecardir.Config[T] (internal/sidecardir) — the single entry for the four families; the version check, the in-memory chain and the canonical serializer belong beside it.
sidecardir.RequireVersion1 — element loading (internal/element/load.go, loadOne) adopts it.
A per-family current version and transform chain, one owner, called from the loader and from the migrate command.
A canonical serializer per family, used by harness init, harness add, harness sidecar and harness migrate; the drafter's # NOTE comment moves to stderr or the harness sidecar output text.
Receipt (internal/addcmd/receipt.go) — the migrate command updates files[].sha256 under the matched-before rule.
cmd/harness — migrate in usage(); the command test ties usage() to docs/harness.md.
internal/checkcmd — the two new warnings.
Starting hints: internal/sidecardir, internal/element/load.go, internal/addcmd, internal/checkcmd, internal/sidecar/generate.go, cmd/harness.

Documentation and skills (same change)

ADR: sidecars are machine-owned, versioned and migrated forward (the decisions above with their reasons).
docs/harness.md: migrate in the command table, a ## Migrate section, the two warnings under "Check", a ### harness migrate --json section, the receipt interaction under "The receipt".
docs/features.md: the receipt hash rule under "Removing" and "Filling gaps", migration of directory bundles under "Bundles from a directory".
scaffold/site/AGENTS.md: the machine-owned header and what it means for hand edits, the meaning of version (bump only on shape change), language files bound to the sidecar's version, harness migrate under the command references.
README.md: one line in the command table, nothing else.
docs/agents/docs-map.md: the new area.
Skills: every skill whose verification runs harness check gains the warning and its fix; at least add-element, add-page-type, add-backend-layout, create-feature (bundle sidecars must be canonical), migrate-site, verify-site, setup-site. scaffold/skills_test.go checks that migrate is a real command.

Acceptance criteria

An element sidecar with version: 2 fails to load with an error naming the file, the same way a page type does today.
harness migrate --site . on a fresh harness init site rewrites nothing and reports unchanged; on a site whose sidecars carry comments or non-canonical key order it rewrites exactly those files, each resulting file loads identically to the original (same Schema value), and a second run reports unchanged.
A migrated language file is rewritten together with its sidecar, and a language file is never rewritten to a version its sidecar does not have.
After harness add news and harness migrate, harness remove news on an otherwise untouched site is not blocked as customised; a vendored file edited by hand before the migrate is still reported customised afterwards.
harness check warns per outdated or non-canonical sidecar and language file, exit 0, and the warning names harness migrate; after harness migrate the warnings are gone.
harness serve opens a site whose sidecars are non-canonical and reports no schema error for them.
harness migrate --json emits exactly the documented object on migrated, unchanged and error, and exits 0, 0 and 1.
A test proves the loader and harness migrate share the chain: a test-only transform registered in the chain changes what the loader reads and what the command writes identically.
Every file harness init, harness add and harness sidecar write is canonical; a drift test fails when a scaffold or embedded bundle sidecar or language file is not canonical or not at the current version.
harness add --dir with a bundle holding a non-canonical sidecar writes the canonical bytes and records their hash.
The documentation and skills listed above are updated; docs/docs_test.go, the command test in cmd/harness, internal/element/agentsdoc_test.go and scaffold/skills_test.go pass.
go test ./... and npm test in web/ pass.

How to verify

go test ./...; cd web && npm test.
In a scratch directory: harness init, harness add news, add a comment to editor/elements/cards.yaml, run harness check (warning), harness migrate (one file, one receipt untouched since cards is not a bundle file), harness check (clean), harness remove news (not blocked).
Edit editor/elements/news-list.yaml by hand, harness migrate, harness remove news reports it customised.

Assumptions to check

No site holds an element sidecar with version above 1; the shipped ones are all 1, production sites cannot be checked from the repository.
sidecardir.Load is the only entry through which the four families are read, including by harness check, harness add and harness remove.
goccy/go-yaml's encoder can be driven to a fixed key order and a header comment deterministically; if not, the serializer writes the document itself.
deckle-hosted reads no sidecar and parses harness.yaml leniently, so dropping comments and reordering keys there needs no hosted change; confirm against deckle-hosted/harness/pin.go.

Out of scope

Any version 2 of any family, and any transform beyond canonicalization.
Content migrations (stored param names or values).
deckle-hosted's trigger, sweep and commit of the migration (its own issue, in that repository).
A warning for untranslated language-file keys.
Palettes, class-wide sidecar overlays and engine capabilities (separate issues).
Preserving a site developer's own comments in sidecars.

Blocked by

None - can start immediately

> *This was generated by AI during triage.* ## Summary Sidecars become machine-owned, versioned files that the harness migrates forward: a `harness migrate` command rewrites every sidecar and its language files to the current version in one canonical form, `harness check` warns about files that need it, and the loaders apply the same migration in memory so an unmigrated site always opens. This is what lets the sidecar schema change after sites are in production without pinning the harness. ## Current behavior - The harness is never pinned (AGENTS.md "Installing the harness"): every site runs the latest release. Nothing today lets a release change the shape of a sidecar, because every sidecar on every customer site would have to be rewritten by hand. - Every sidecar file carries `version`. Page types, backend layouts and editor modules pin it to exactly 1 through `sidecardir.RequireVersion1`; element sidecars (`internal/element/load.go`) accept any integer `>= 1` and ignore it. Every shipped sidecar is version 1. - Sidecars are hand-formatted YAML with long explanatory comments (`scaffold/editor/elements/*.yaml`, the bundles). The sidecar drafter (`internal/sidecar`) writes a `# NOTE` comment in one case. - Language files (`<name>.<code>.yaml`, "Sidecar language files" in AGENTS.md) carry no version; they mirror the sidecar's keys and are read only after their sidecar loaded. - A feature receipt (`editor/features/<name>.yaml`, docs/harness.md "The receipt") records the SHA-256 of every vendored file as written by `harness add`. Nothing ever re-derives it; a file whose bytes differ counts as customised, which makes the feature "in use" and blocks `harness remove` behind `--purge`. - `harness check` already warns about a content file that is not canonical; there is no equivalent for sidecars. - deckle-hosted runs `harness serve` directly on a site's worktree and never runs `harness check` before opening the editor. It parses `harness.yaml` and `feature.yaml`, never sidecars. ## Desired behavior **Versions.** Each sidecar family has a current version the harness knows. A file at the current version loads as today. A file at an older supported version loads through the migration chain in memory, in every loader (`harness serve`, `harness check`, `harness add`, …), so no release ever locks a site out of its editor. A file above the current version, or below the oldest supported one, is a load error naming the file and the version. **Canonical form.** The harness owns the sidecar's serialization the way the content engine owns the content format (ADR 0001): fixed key order, one spelling per value, no comments, and a header comment the serializer itself emits saying the file is machine-owned by the harness and that comments do not survive. Every sidecar the harness writes (`harness init`, `harness add`, `harness sidecar`, `harness migrate`) is canonical. The shipped scaffold and the embedded bundles are canonical and at the current version, enforced by a drift test. **`harness migrate [--site <dir>] [--json]`.** Rewrites every sidecar of every family (elements, page types, backend layouts, editor modules) and each sidecar's language files to the current version in canonical form; a file already canonical and current is not touched. For every rewritten file listed in a feature receipt whose bytes matched the receipt's hash before the rewrite, the receipt's hash is updated; a file that did not match is left as customised. Exit 0 whether or not anything changed; exit 1 when a file cannot be migrated (it does not load even through the chain, or cannot be written), with every other file still migrated. Runs without Hugo, like `harness add`, so a host can run it in a clean clone. `--json` follows the add command's conventions: `{"formatVersion": 1, "status": "migrated" | "unchanged", "files": [paths rewritten], "receipts": [receipt paths rewritten]}`, and `{"formatVersion": 1, "status": "error", "error": "…", "files": […]}` on exit 1. **`harness check`.** Warns, per file, about a sidecar or language file below the current version or not in canonical form, naming `harness migrate` as the fix. Both are warnings: the file still loads. **Loader and chain.** One chain of per-version transforms per family, each a pure function from the raw document of version N and its language files to version N+1. The loader applies it in memory and `harness migrate` applies it to disk; nothing else reads an old version. Today the chain holds no transform (there is no version 2); the structure, its tests and the two call sites are the deliverable, with the canonicalization of a version 1 file as the first real rewrite. **Directory bundles.** `harness add --dir` migrates a bundle's sidecars and language files while copying them, and the receipt records the bytes written. ## Decisions - The harness stays unpinned and sidecars migrate forward — one harness version in production is the security stance; a pin would spread sites across releases. - The version is per file, not per site — a file stays self-describing, bundles vendor files, and a directory bundle may arrive at an older version. - Element sidecars are pinned to version 1 through `sidecardir.RequireVersion1`, the helper the other three families use — one owner of the rule, and a site must never hold a meaningless version 2 before version 2 has a meaning. - A bump means a shape change that needs a rewrite; a new optional key never bumps — the loader reads version N with or without the key, and a bump forces a commit on every customer repository. - Migrations change shape, never stored param names or value spellings — renaming a param is a content migration across every page of every site, a different and heavier thing (the category rename path is the precedent), and keeping the two apart is what makes sidecar migrations cheap enough to ship. - The transform is one function with two callers (loader in memory, `migrate` on disk) — "Every rule has exactly one owner" (CLAUDE.md); a loader that read old versions its own way would be a second copy. - Every version back to 1 stays supported; dropping one is a deliberate decision, never a side effect. - Sidecars are machine-owned and comments are dropped — the facts belong in AGENTS.md ("Every fact has one document"), and a comment-preserving migration would have to be a textual edit per version, the cost that keeps migrations rare. The header comment is the one comment, emitted by the serializer. - Language files carry no version; they are bound to their sidecar's — their keys mirror the sidecar's keys, so a transform that renames a structural key renames it in both, and the loader already reads a language file only after its sidecar loaded. - Receipt hashes are rewritten only for files whose bytes matched the receipt before migration — a customised file stays customised, which is what the hash exists to say; without the rewrite every bundle feature on every site would need `--purge` to remove after one migration, and the gap fill never overwrites a file, so the hash would never catch up. - Outdated and non-canonical are warnings, not errors — the file loads. - deckle-hosted runs `harness migrate` per site on each harness upgrade, commits to main and reconciles the editor branch, through the path a feature add already uses; that is a deckle-hosted issue. The `--json` shape here is the cross-repo contract it consumes, hence `formatVersion`. ## Key interfaces - `sidecardir.Load` and `sidecardir.Config[T]` (`internal/sidecardir`) — the single entry for the four families; the version check, the in-memory chain and the canonical serializer belong beside it. - `sidecardir.RequireVersion1` — element loading (`internal/element/load.go`, `loadOne`) adopts it. - A per-family current version and transform chain, one owner, called from the loader and from the migrate command. - A canonical serializer per family, used by `harness init`, `harness add`, `harness sidecar` and `harness migrate`; the drafter's `# NOTE` comment moves to stderr or the `harness sidecar` output text. - `Receipt` (`internal/addcmd/receipt.go`) — the migrate command updates `files[].sha256` under the matched-before rule. - `cmd/harness` — `migrate` in `usage()`; the command test ties `usage()` to docs/harness.md. - `internal/checkcmd` — the two new warnings. - Starting hints: `internal/sidecardir`, `internal/element/load.go`, `internal/addcmd`, `internal/checkcmd`, `internal/sidecar/generate.go`, `cmd/harness`. ## Documentation and skills (same change) - ADR: sidecars are machine-owned, versioned and migrated forward (the decisions above with their reasons). - `docs/harness.md`: `migrate` in the command table, a `## Migrate` section, the two warnings under "Check", a `### harness migrate --json` section, the receipt interaction under "The receipt". - `docs/features.md`: the receipt hash rule under "Removing" and "Filling gaps", migration of directory bundles under "Bundles from a directory". - `scaffold/site/AGENTS.md`: the machine-owned header and what it means for hand edits, the meaning of `version` (bump only on shape change), language files bound to the sidecar's version, `harness migrate` under the command references. - `README.md`: one line in the command table, nothing else. - `docs/agents/docs-map.md`: the new area. - Skills: every skill whose verification runs `harness check` gains the warning and its fix; at least `add-element`, `add-page-type`, `add-backend-layout`, `create-feature` (bundle sidecars must be canonical), `migrate-site`, `verify-site`, `setup-site`. `scaffold/skills_test.go` checks that `migrate` is a real command. ## Acceptance criteria - [ ] An element sidecar with `version: 2` fails to load with an error naming the file, the same way a page type does today. - [ ] `harness migrate --site .` on a fresh `harness init` site rewrites nothing and reports `unchanged`; on a site whose sidecars carry comments or non-canonical key order it rewrites exactly those files, each resulting file loads identically to the original (same `Schema` value), and a second run reports `unchanged`. - [ ] A migrated language file is rewritten together with its sidecar, and a language file is never rewritten to a version its sidecar does not have. - [ ] After `harness add news` and `harness migrate`, `harness remove news` on an otherwise untouched site is not blocked as customised; a vendored file edited by hand before the migrate is still reported customised afterwards. - [ ] `harness check` warns per outdated or non-canonical sidecar and language file, exit 0, and the warning names `harness migrate`; after `harness migrate` the warnings are gone. - [ ] `harness serve` opens a site whose sidecars are non-canonical and reports no schema error for them. - [ ] `harness migrate --json` emits exactly the documented object on `migrated`, `unchanged` and `error`, and exits 0, 0 and 1. - [ ] A test proves the loader and `harness migrate` share the chain: a test-only transform registered in the chain changes what the loader reads and what the command writes identically. - [ ] Every file `harness init`, `harness add` and `harness sidecar` write is canonical; a drift test fails when a scaffold or embedded bundle sidecar or language file is not canonical or not at the current version. - [ ] `harness add --dir` with a bundle holding a non-canonical sidecar writes the canonical bytes and records their hash. - [ ] The documentation and skills listed above are updated; `docs/docs_test.go`, the command test in `cmd/harness`, `internal/element/agentsdoc_test.go` and `scaffold/skills_test.go` pass. - [ ] `go test ./...` and `npm test` in `web/` pass. ## How to verify - `go test ./...`; `cd web && npm test`. - In a scratch directory: `harness init`, `harness add news`, add a comment to `editor/elements/cards.yaml`, run `harness check` (warning), `harness migrate` (one file, one receipt untouched since cards is not a bundle file), `harness check` (clean), `harness remove news` (not blocked). - Edit `editor/elements/news-list.yaml` by hand, `harness migrate`, `harness remove news` reports it customised. ## Assumptions to check - No site holds an element sidecar with `version` above 1; the shipped ones are all 1, production sites cannot be checked from the repository. - `sidecardir.Load` is the only entry through which the four families are read, including by `harness check`, `harness add` and `harness remove`. - goccy/go-yaml's encoder can be driven to a fixed key order and a header comment deterministically; if not, the serializer writes the document itself. - deckle-hosted reads no sidecar and parses `harness.yaml` leniently, so dropping comments and reordering keys there needs no hosted change; confirm against `deckle-hosted/harness/pin.go`. ## Out of scope - Any version 2 of any family, and any transform beyond canonicalization. - Content migrations (stored param names or values). - deckle-hosted's trigger, sweep and commit of the migration (its own issue, in that repository). - A warning for untranslated language-file keys. - Palettes, class-wide sidecar overlays and engine capabilities (separate issues). - Preserving a site developer's own comments in sidecars. ## Blocked by None - can start immediately

dominik.polakovics added the

enhancement

ready-for-agent

labels

2026-10-15 12:26:59 +02:00

dominik.polakovics referenced this issue

2026-10-15 12:27:55 +02:00

Engine capabilities: feature-toggled Access-tab fields, first scheduled visibility of elements #303

dominik.polakovics referenced this issue

2026-10-15 12:27:55 +02:00

Class-wide additive sidecar overlays from feature bundles, with a theme hook registry #304

dominik.polakovics referenced this issue