feat(harness): harness bench measures whether the templates build linearly in the page count #367

Open
dominik.polakovics wants to merge 3 commits from afk/363 into main

Closes #363

What

  • internal/sitebench (new) owns the rule "does this theme build linearly in the page count". It holds the news-item generator (Items, SiteContent), the one-core build runner (RunBuild: GOMAXPROCS=1, a private HUGO_CACHEDIR, killed when its context ends), the template-metrics parser (both the plain and the hints table), MaxRatio = 2.7 with its calibration comment, and Measure, which builds N, 2N, N, 2N and compares the fastest build of each size. Run benches a real site: it checks the news receipt, resolves the section (the first section of type news by default, or --section), copies the site twice into a temporary directory, fills each copy with generated items, measures, and optionally runs one time-boxed hints build. The site is never written.
  • internal/theme/scale_test.go calls the package and has no generator, runner, parser or limit of its own. TestBuildTimeScalesLinearly keeps its behaviour: N = 2,000, limit 2.7, plain timed builds that must be clean, and the once-per-key templateMetrics subtest.
  • harness bench [--site <dir>] --pages <n> [--section <dir>] [--hints] [--timeout <d>] [--json]:
    • Exits 0 on ok, 1 on slow, and 2 when it cannot run: no hugo, no news bundle, no section, or a failed build (Hugo's output goes to stderr).
    • --json prints one object: formatVersion, status, section, builds, ratio, limit, templates, plus hints and cachePotential only with --hints.
    • Hints are opt-in and time-boxed because Hugo's hints mode can deadlock. When the hints build times out, the verdict from the plain builds still stands.
    • serve and bench now share one Hugo-resolution helper (resolveHugo).
  • Docs, each fact in the document that owns it:
    • docs/harness.md: a "Bench" section and "harness bench --json".
    • README.md: one command-table row.
    • scaffold/site/AGENTS.md: the "Verifying a change" paragraph.
    • scaffold/themes/deckle/README.md "Navigation": one sentence.
    • docs/agents/docs-map.md: the new package.
  • Skills:
    • verify-site gets a new step 3 that runs harness bench --site . --pages 2500 after a change to a template that lists pages, and stops on slow.
    • redo-theme gets a step before Verify that points at it. migrate-site (steps 7 and 9), create-feature and add-page-type name the bench where they point at verify-site.
  • scaffold/skills_test.go now reads the harness command set from the usage block in docs/harness.md, which cmd/harness's docs test already pins to the commands table. It used to keep a hand copy, which CLAUDE.md calls a second owner.
  • harness check is unchanged.

Verification

  • go vet ./..., go test ./... and gofmt are all clean locally.
  • cmd/harness/bench_test.go drives the command against a stub hugo on PATH. It covers:
    • ok and slow verdicts
    • the strict --json shape
    • --hints with cachePotential
    • a hints timeout with --timeout 1s
    • no hugo on PATH, no news feature, and no news section
    • a failed build
    • usage errors
    • the site staying byte-unchanged
  • internal/sitebench tests cover:
    • the generator: canonical round trip, and distinct past dates up to 20,000 items
    • the parser, on both table forms
    • RunBuild against a stub: environment, kill on deadline, and the BuildError output
    • Run with an injected runner: section resolution, temp cleanup, refusing to overwrite, and the hints paths

Not run locally: this machine has no Hugo, and downloading it was not permitted in this run. So TestBuildTimeScalesLinearly skipped locally and runs only in CI, which installs Hugo 0.167.0. The acceptance runs against real Hugo were not done:

  • the fixed theme vs. the pre-#364 partials at --pages 500 and --pages 2000
  • the 2,534-item hints deadlock

They are worth doing by hand before merging.

Notes

  • Generated items are named <section>/bench-item-NNNNN/_index.md. Their dates go back one hour per item from 2026-01-01, so they stay in the past for any realistic N, even on a site without buildFuture. The guard's generated site changes the same way; its shape is unchanged.
  • The page count is summed over every language column of Hugo's summary. The old regex read only the first column.

🤖 Generated with Claude Code

Closes #363 ## What - **`internal/sitebench`** (new) owns the rule "does this theme build linearly in the page count". It holds the news-item generator (`Items`, `SiteContent`), the one-core build runner (`RunBuild`: `GOMAXPROCS=1`, a private `HUGO_CACHEDIR`, killed when its context ends), the template-metrics parser (both the plain and the hints table), `MaxRatio` = 2.7 with its calibration comment, and `Measure`, which builds N, 2N, N, 2N and compares the fastest build of each size. `Run` benches a real site: it checks the news receipt, resolves the section (the first section of type `news` by default, or `--section`), copies the site twice into a temporary directory, fills each copy with generated items, measures, and optionally runs one time-boxed hints build. The site is never written. - **`internal/theme/scale_test.go`** calls the package and has no generator, runner, parser or limit of its own. `TestBuildTimeScalesLinearly` keeps its behaviour: N = 2,000, limit 2.7, plain timed builds that must be clean, and the once-per-key `templateMetrics` subtest. - **`harness bench [--site <dir>] --pages <n> [--section <dir>] [--hints] [--timeout <d>] [--json]`**: - Exits 0 on `ok`, 1 on `slow`, and 2 when it cannot run: no hugo, no news bundle, no section, or a failed build (Hugo's output goes to stderr). - `--json` prints one object: `formatVersion`, `status`, `section`, `builds`, `ratio`, `limit`, `templates`, plus `hints` and `cachePotential` only with `--hints`. - Hints are opt-in and time-boxed because Hugo's hints mode can deadlock. When the hints build times out, the verdict from the plain builds still stands. - `serve` and `bench` now share one Hugo-resolution helper (`resolveHugo`). - **Docs**, each fact in the document that owns it: - `docs/harness.md`: a "Bench" section and "harness bench --json". - `README.md`: one command-table row. - `scaffold/site/AGENTS.md`: the "Verifying a change" paragraph. - `scaffold/themes/deckle/README.md` "Navigation": one sentence. - `docs/agents/docs-map.md`: the new package. - **Skills**: - `verify-site` gets a new step 3 that runs `harness bench --site . --pages 2500` after a change to a template that lists pages, and stops on `slow`. - `redo-theme` gets a step before Verify that points at it. `migrate-site` (steps 7 and 9), `create-feature` and `add-page-type` name the bench where they point at verify-site. - **`scaffold/skills_test.go`** now reads the harness command set from the usage block in `docs/harness.md`, which `cmd/harness`'s docs test already pins to the commands table. It used to keep a hand copy, which CLAUDE.md calls a second owner. - `harness check` is unchanged. ## Verification - `go vet ./...`, `go test ./...` and gofmt are all clean locally. - `cmd/harness/bench_test.go` drives the command against a stub `hugo` on PATH. It covers: - ok and slow verdicts - the strict `--json` shape - `--hints` with `cachePotential` - a hints timeout with `--timeout 1s` - no hugo on PATH, no news feature, and no news section - a failed build - usage errors - the site staying byte-unchanged - `internal/sitebench` tests cover: - the generator: canonical round trip, and distinct past dates up to 20,000 items - the parser, on both table forms - `RunBuild` against a stub: environment, kill on deadline, and the `BuildError` output - `Run` with an injected runner: section resolution, temp cleanup, refusing to overwrite, and the hints paths **Not run locally:** this machine has no Hugo, and downloading it was not permitted in this run. So `TestBuildTimeScalesLinearly` skipped locally and runs only in CI, which installs Hugo 0.167.0. The acceptance runs against real Hugo were not done: - the fixed theme vs. the pre-#364 partials at `--pages 500` and `--pages 2000` - the 2,534-item hints deadlock They are worth doing by hand before merging. ## Notes - Generated items are named `<section>/bench-item-NNNNN/_index.md`. Their dates go back one hour per item from 2026-01-01, so they stay in the past for any realistic N, even on a site without `buildFuture`. The guard's generated site changes the same way; its shape is unchanged. - The page count is summed over every language column of Hugo's summary. The old regex read only the first column. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
internal/sitebench owns the rule "does this theme build linearly in the page
count": the news-item generator (Items, SiteContent), the one-core build
runner (RunBuild), the template-metrics parser, MaxRatio with its
calibration, and Measure (interleaved best-of-N). Run benches a real site on
copies with a generated news section. The theme guard test now calls the
package and keeps its assertions.

Refs #363

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
feat(harness): harness bench measures whether the templates build linearly
Some checks failed
ci / build-and-test (pull_request) Failing after 4m20s
ci / release (pull_request) Has been skipped
25e2376e11
`harness bench --site <dir> --pages <n> [--section <dir>] [--hints]
[--timeout <d>] [--json]` builds copies of the site at n and 2n generated
news items on one core with the site's Hugo and reports both builds, the
ratio against sitebench.MaxRatio, and the top template rows; exit 0 ok,
1 slow, 2 cannot run. Hints are opt-in and time-boxed because Hugo's hints
mode can deadlock. serve and bench share one Hugo resolution helper.

verify-site gains a bench step for changes to templates that list pages;
redo-theme, migrate-site, create-feature and add-page-type point at it.
AGENTS.md "Verifying a change", the theme README "Navigation",
docs/harness.md, the README command table and the docs map name the bench.
The skills drift test reads the command set from docs/harness.md instead
of a hand copy.

Closes #363

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Author
Owner

[autoland] verdict: reject

Verdict: FAIL

Signal relied on: labctl pr checks 367 → failure (ci / build-and-test, run 399, Hugo 0.167.0). labctl pr logs could not fetch the log because the adapter does not match this forge version. I read the job log at /Cloonar/deckle/actions/runs/399/jobs/0/attempt/1/logs and reproduced the cause locally.

Blocker: ParseTemplateMetrics parses no rows from real Hugo output

go test ./... fails in internal/theme:

--- FAIL: TestBuildTimeScalesLinearly/templateMetrics
    scale_test.go:129: template metrics have no row for footer-pages.html; the table hugo printed: …
    scale_test.go:129: template metrics have no row for news-items.html; …

Both rows are in the table Hugo printed (_partials/footer-pages.html … 2487 2488, _partials/news-items.html … 199 200). The parser cannot read them. Hugo 0.167 prints the durations as a number, space(s), then the unit:

     16.64  s       7.52 ms     201.12 ms          0        0       0   2213  list.html
  • internal/sitebench/build.go:176 documents the durations as Go duration strings ("1.234567891s"). They are not.
  • internal/sitebench/build.go:208 (parseRow) runs time.ParseDuration on f[0..2] after strings.Fields. That gets "16.64", "s", "7.52", … and rejects every row.

I fed the CI table to the PR's ParseTemplateMetrics locally and got hints=true rows=0.

This is a regression. Main's parseTemplateMetrics (the old internal/theme/scale_test.go:240) read cached/total/template from the end of each row and never parsed the durations, so it worked on this output.

What it breaks beyond the test

  • harness bench gets Templates from this parser (internal/sitebench/bench.go:178, and :199 with --hints). On real Hugo, --json templates is always [] and the cachePotential column never appears. The text output silently drops the "where to look" table (cmd/harness/bench.go:257).
  • Issue #363 requires both of these: the top metrics rows ("template, count, cumulative, average, so the developer knows where to look"), and that the guard test keeps its once-per-key subtest. Only the ratio verdict works: CI measured 1.78 against the 2.7 limit.

Why the unit tests did not catch it

The fixtures were hand-written in Go duration format, not taken from Hugo:

  • internal/sitebench/build_test.go:20 and :31
  • cmd/harness/bench_test.go:75 and :84 (the stub hugo)

The PR body notes Hugo was not available locally, so the only real-Hugo run was in CI, and that is where it failed.

What to fix

  1. Parse Hugo's real duration format. Join each number with the unit after it (16.64 s, 7.52 ms, 201.12 ms, 63.28 µs) before parsing, or parse the counts and template from the end of the row as main did. Keep handling both table forms.
  2. Use real Hugo output in the tests. Replace the fixtures in internal/sitebench/build_test.go and the stub tables in cmd/harness/bench_test.go with a table captured from Hugo 0.167. The CI log of run 399 has a complete hints table. Add a plain --templateMetrics table too, so a format drift fails without Hugo.
  3. Re-run go test ./internal/theme with Hugo on PATH, or let CI do it, and confirm templateMetrics passes.

Checked and fine

  • Title is Conventional Commits.
  • Closes #363 is present.
  • Diff scope matches the issue. The scaffold/skills_test.go change follows the one-owner rule.
  • Skills, AGENTS.md, theme README and docs/harness.md are updated per CLAUDE.md.
  • Every other package passed in CI, including internal/sitebench, cmd/harness, docs and scaffold.
[autoland] verdict: reject ## Verdict: FAIL **Signal relied on:** `labctl pr checks 367` → `failure` (`ci / build-and-test`, run 399, Hugo 0.167.0). `labctl pr logs` could not fetch the log because the adapter does not match this forge version. I read the job log at `/Cloonar/deckle/actions/runs/399/jobs/0/attempt/1/logs` and reproduced the cause locally. ### Blocker: `ParseTemplateMetrics` parses no rows from real Hugo output `go test ./...` fails in `internal/theme`: ``` --- FAIL: TestBuildTimeScalesLinearly/templateMetrics scale_test.go:129: template metrics have no row for footer-pages.html; the table hugo printed: … scale_test.go:129: template metrics have no row for news-items.html; … ``` Both rows are in the table Hugo printed (`_partials/footer-pages.html … 2487 2488`, `_partials/news-items.html … 199 200`). The parser cannot read them. Hugo 0.167 prints the durations as a number, space(s), then the unit: ``` 16.64 s 7.52 ms 201.12 ms 0 0 0 2213 list.html ``` - `internal/sitebench/build.go:176` documents the durations as Go duration strings (`"1.234567891s"`). They are not. - `internal/sitebench/build.go:208` (`parseRow`) runs `time.ParseDuration` on `f[0..2]` after `strings.Fields`. That gets `"16.64"`, `"s"`, `"7.52"`, … and rejects every row. I fed the CI table to the PR's `ParseTemplateMetrics` locally and got `hints=true rows=0`. This is a regression. Main's `parseTemplateMetrics` (the old `internal/theme/scale_test.go:240`) read `cached`/`total`/`template` from the end of each row and never parsed the durations, so it worked on this output. ### What it breaks beyond the test - `harness bench` gets `Templates` from this parser (`internal/sitebench/bench.go:178`, and `:199` with `--hints`). On real Hugo, `--json` `templates` is always `[]` and the `cachePotential` column never appears. The text output silently drops the "where to look" table (`cmd/harness/bench.go:257`). - Issue #363 requires both of these: the top metrics rows ("template, count, cumulative, average, so the developer knows where to look"), and that the guard test keeps its once-per-key subtest. Only the ratio verdict works: CI measured 1.78 against the 2.7 limit. ### Why the unit tests did not catch it The fixtures were hand-written in Go duration format, not taken from Hugo: - `internal/sitebench/build_test.go:20` and `:31` - `cmd/harness/bench_test.go:75` and `:84` (the stub hugo) The PR body notes Hugo was not available locally, so the only real-Hugo run was in CI, and that is where it failed. ### What to fix 1. **Parse Hugo's real duration format.** Join each number with the unit after it (`16.64 s`, `7.52 ms`, `201.12 ms`, `63.28 µs`) before parsing, or parse the counts and template from the end of the row as main did. Keep handling both table forms. 2. **Use real Hugo output in the tests.** Replace the fixtures in `internal/sitebench/build_test.go` and the stub tables in `cmd/harness/bench_test.go` with a table captured from Hugo 0.167. The CI log of run 399 has a complete hints table. Add a plain `--templateMetrics` table too, so a format drift fails without Hugo. 3. **Re-run `go test ./internal/theme`** with Hugo on PATH, or let CI do it, and confirm `templateMetrics` passes. ### Checked and fine - Title is Conventional Commits. - `Closes #363` is present. - Diff scope matches the issue. The `scaffold/skills_test.go` change follows the one-owner rule. - Skills, AGENTS.md, theme README and `docs/harness.md` are updated per CLAUDE.md. - Every other package passed in CI, including `internal/sitebench`, `cmd/harness`, `docs` and `scaffold`.
fix(sitebench): read Hugo 0.167's template metrics durations
All checks were successful
ci / build-and-test (pull_request) Successful in 4m23s
ci / release (pull_request) Has been skipped
6b1f7fb09a
Hugo 0.167 prints each duration in the --templateMetrics table as a
number and a unit in separate fields ("16.64  s", "7.52 ms",
"63.28 µs"), not as a Go duration string. parseRow ran
time.ParseDuration on the bare number and rejected every row, so
harness bench reported no templates and the guard test's
once-per-key subtest found no footer-pages.html or news-items.html.

parseDuration now joins a number with the unit field after it, and
still reads a single Go duration string (Hugo 0.161's form).

The parser tests read testdata/hugo-0.167-hints.txt, the table Hugo
0.167.0 printed in CI run 399, and a plain-layout table with the same
rows. The stub hugo in cmd/harness prints those files, and the fakes
in the sitebench tests print Hugo 0.167's format.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Author
Owner

[autoland] verdict: fix-done

[autoland] verdict: fix-done
Author
Owner

[autoland] verdict: pass

Verdict: CONCERNS (approved, not merged)

Signal relied on: labctl pr checks 367 --wait → success, from CI run 400 on head 6b1f7fb. I read the job log (/Cloonar/deckle/actions/runs/400/jobs/0/attempt/1/logs). Hugo 0.167.0 extended was installed. go vet ./... and go test ./... were green. internal/theme ran for 50.3 s, so TestBuildTimeScalesLinearly and its templateMetrics subtest ran against real Hugo rather than skipping. internal/sitebench, cmd/harness, docs and scaffold all passed.

The previous rejection is resolved

  • internal/sitebench/build.go parseDuration now joins a number with the unit field that follows it.
  • I checked it against Hugo v0.167.0's own metrics/metrics.go:
    • formatDuration prints %.2f s, %.2f ms, %.2f µs and %.2f ns. All four units are in durationUnits.
    • The plain row is %15s %12s %12s %5d %s (sum, avg, max, count, key).
    • The hints row adds %9d %7.f %6d %5d.
    • Both match the parser.
  • So the hand-built testdata/hugo-0.167-plain.txt has the same shape as real Hugo output. The hints fixture was captured from CI run 399.

Concern (non-blocking): the real-Hugo acceptance criteria were never run

CI exercises real Hugo only through the guard test, which covers Measure, RunBuild and the hints-form parser on the shipped theme. It does not cover the following, and the PR body says they are worth doing by hand before merging:

  • Criterion 1: harness bench --pages 500 on a fresh site with news and categories. This exercises sitebench.Run end to end: copySite, resolveSection, and Items written into a real news section. The default plain-table path is also only checked against real Hugo by the source comparison above.
  • Criterion 2: old partials at --pages 2000 → slow and exit 1; the fixed theme → ok. Unlike the guard, the bench passes --templateMetrics on every timed build, so this confirms the 2.7 limit still separates the two with that overhead included.
  • Criterion 4: --hints on the fixed site shows cachePotential, and --timeout 1s reproduces the hints timeout against old partials at 2,534 items.

The last round showed that stub-only coverage can miss a real-Hugo-only bug. A maintainer should run these three by hand before merging. The lander could not, because downloading Hugo was not permitted in this run.

Checked and fine

  • Title is Conventional Commits. Closes #363 is present.
  • Diff scope matches the issue's Desired behavior and Key interfaces.
    • The resolveHugo extraction from serve is what lets the bench use the pinned Hugo.
    • scaffold/skills_test.go now reads the command set from docs/harness.md, which removes a hand copy (the one-owner rule).
  • Docs: each new fact is in the document that owns it. That covers the docs/harness.md "Bench" and "harness bench --json" sections, the one README row, the AGENTS.md "Verifying a change" paragraph, one theme README "Navigation" sentence, and the docs-map entry.
  • Skills: verify-site has the new step 3. redo-theme has the new step 4, and the later steps were renumbered; I checked that no step reference elsewhere is left stale. migrate-site (steps 7 and 9), create-feature and add-page-type point at verify-site without restating the thresholds.
  • harness check is unchanged.
[autoland] verdict: pass ## Verdict: CONCERNS (approved, not merged) **Signal relied on:** `labctl pr checks 367 --wait` → `success`, from CI run 400 on head `6b1f7fb`. I read the job log (`/Cloonar/deckle/actions/runs/400/jobs/0/attempt/1/logs`). Hugo 0.167.0 extended was installed. `go vet ./...` and `go test ./...` were green. `internal/theme` ran for 50.3 s, so `TestBuildTimeScalesLinearly` and its `templateMetrics` subtest ran against real Hugo rather than skipping. `internal/sitebench`, `cmd/harness`, `docs` and `scaffold` all passed. ### The previous rejection is resolved - `internal/sitebench/build.go` `parseDuration` now joins a number with the unit field that follows it. - I checked it against Hugo v0.167.0's own `metrics/metrics.go`: - `formatDuration` prints `%.2f s`, `%.2f ms`, `%.2f µs` and `%.2f ns`. All four units are in `durationUnits`. - The plain row is `%15s %12s %12s %5d %s` (sum, avg, max, count, key). - The hints row adds `%9d %7.f %6d %5d`. - Both match the parser. - So the hand-built `testdata/hugo-0.167-plain.txt` has the same shape as real Hugo output. The hints fixture was captured from CI run 399. ### Concern (non-blocking): the real-Hugo acceptance criteria were never run CI exercises real Hugo only through the guard test, which covers `Measure`, `RunBuild` and the hints-form parser on the shipped theme. It does not cover the following, and the PR body says they are worth doing by hand before merging: - **Criterion 1:** `harness bench --pages 500` on a fresh site with news and categories. This exercises `sitebench.Run` end to end: `copySite`, `resolveSection`, and `Items` written into a real news section. The default plain-table path is also only checked against real Hugo by the source comparison above. - **Criterion 2:** old partials at `--pages 2000` → `slow` and exit 1; the fixed theme → `ok`. Unlike the guard, the bench passes `--templateMetrics` on every timed build, so this confirms the 2.7 limit still separates the two with that overhead included. - **Criterion 4:** `--hints` on the fixed site shows `cachePotential`, and `--timeout 1s` reproduces the hints timeout against old partials at 2,534 items. The last round showed that stub-only coverage can miss a real-Hugo-only bug. A maintainer should run these three by hand before merging. The lander could not, because downloading Hugo was not permitted in this run. ### Checked and fine - **Title** is Conventional Commits. **`Closes #363`** is present. - **Diff scope** matches the issue's Desired behavior and Key interfaces. - The `resolveHugo` extraction from `serve` is what lets the bench use the pinned Hugo. - `scaffold/skills_test.go` now reads the command set from `docs/harness.md`, which removes a hand copy (the one-owner rule). - **Docs:** each new fact is in the document that owns it. That covers the `docs/harness.md` "Bench" and "harness bench --json" sections, the one README row, the AGENTS.md "Verifying a change" paragraph, one theme README "Navigation" sentence, and the docs-map entry. - **Skills:** `verify-site` has the new step 3. `redo-theme` has the new step 4, and the later steps were renumbered; I checked that no step reference elsewhere is left stale. `migrate-site` (steps 7 and 9), `create-feature` and `add-page-type` point at verify-site without restating the thresholds. - **`harness check`** is unchanged.
All checks were successful
ci / build-and-test (pull_request) Successful in 4m23s
ci / release (pull_request) Has been skipped
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin afk/363:afk/363
git switch afk/363

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff afk/363
git switch afk/363
git rebase main
git switch main
git merge --ff-only afk/363
git switch afk/363
git rebase main
git switch main
git merge --no-ff afk/363
git switch main
git merge --squash afk/363
git switch main
git merge --ff-only afk/363
git switch main
git merge afk/363
git push origin main
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Cloonar/deckle!367
No description provided.