feat(harness): harness bench measures whether the templates build linearly in the page count #367
No reviewers
Labels
No labels
bug
enhancement
needs-triage
ready-for-agent
ready-for-human
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Cloonar/deckle!367
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "afk/363"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #363
What
internal/sitebench(new) owns the rule "does this theme build linearly in the page count". It holds the news-item generator (Items,SiteContent), the one-core build runner (RunBuild:GOMAXPROCS=1, a privateHUGO_CACHEDIR, killed when its context ends), the template-metrics parser (both the plain and the hints table),MaxRatio= 2.7 with its calibration comment, andMeasure, which builds N, 2N, N, 2N and compares the fastest build of each size.Runbenches a real site: it checks the news receipt, resolves the section (the first section of typenewsby default, or--section), copies the site twice into a temporary directory, fills each copy with generated items, measures, and optionally runs one time-boxed hints build. The site is never written.internal/theme/scale_test.gocalls the package and has no generator, runner, parser or limit of its own.TestBuildTimeScalesLinearlykeeps its behaviour: N = 2,000, limit 2.7, plain timed builds that must be clean, and the once-per-keytemplateMetricssubtest.harness bench [--site <dir>] --pages <n> [--section <dir>] [--hints] [--timeout <d>] [--json]:ok, 1 onslow, and 2 when it cannot run: no hugo, no news bundle, no section, or a failed build (Hugo's output goes to stderr).--jsonprints one object:formatVersion,status,section,builds,ratio,limit,templates, plushintsandcachePotentialonly with--hints.serveandbenchnow share one Hugo-resolution helper (resolveHugo).docs/harness.md: a "Bench" section and "harness bench --json".README.md: one command-table row.scaffold/site/AGENTS.md: the "Verifying a change" paragraph.scaffold/themes/deckle/README.md"Navigation": one sentence.docs/agents/docs-map.md: the new package.verify-sitegets a new step 3 that runsharness bench --site . --pages 2500after a change to a template that lists pages, and stops onslow.redo-themegets a step before Verify that points at it.migrate-site(steps 7 and 9),create-featureandadd-page-typename the bench where they point at verify-site.scaffold/skills_test.gonow reads the harness command set from the usage block indocs/harness.md, whichcmd/harness's docs test already pins to the commands table. It used to keep a hand copy, which CLAUDE.md calls a second owner.harness checkis unchanged.Verification
go vet ./...,go test ./...and gofmt are all clean locally.cmd/harness/bench_test.godrives the command against a stubhugoon PATH. It covers:--jsonshape--hintswithcachePotential--timeout 1sinternal/sitebenchtests cover:RunBuildagainst a stub: environment, kill on deadline, and theBuildErroroutputRunwith an injected runner: section resolution, temp cleanup, refusing to overwrite, and the hints pathsNot run locally: this machine has no Hugo, and downloading it was not permitted in this run. So
TestBuildTimeScalesLinearlyskipped locally and runs only in CI, which installs Hugo 0.167.0. The acceptance runs against real Hugo were not done:--pages 500and--pages 2000They are worth doing by hand before merging.
Notes
<section>/bench-item-NNNNN/_index.md. Their dates go back one hour per item from 2026-01-01, so they stay in the past for any realistic N, even on a site withoutbuildFuture. The guard's generated site changes the same way; its shape is unchanged.🤖 Generated with Claude Code
[autoland] verdict: reject
Verdict: FAIL
Signal relied on:
labctl pr checks 367→failure(ci / build-and-test, run 399, Hugo 0.167.0).labctl pr logscould not fetch the log because the adapter does not match this forge version. I read the job log at/Cloonar/deckle/actions/runs/399/jobs/0/attempt/1/logsand reproduced the cause locally.Blocker:
ParseTemplateMetricsparses no rows from real Hugo outputgo test ./...fails ininternal/theme:Both rows are in the table Hugo printed (
_partials/footer-pages.html … 2487 2488,_partials/news-items.html … 199 200). The parser cannot read them. Hugo 0.167 prints the durations as a number, space(s), then the unit:internal/sitebench/build.go:176documents the durations as Go duration strings ("1.234567891s"). They are not.internal/sitebench/build.go:208(parseRow) runstime.ParseDurationonf[0..2]afterstrings.Fields. That gets"16.64","s","7.52", … and rejects every row.I fed the CI table to the PR's
ParseTemplateMetricslocally and gothints=true rows=0.This is a regression. Main's
parseTemplateMetrics(the oldinternal/theme/scale_test.go:240) readcached/total/templatefrom the end of each row and never parsed the durations, so it worked on this output.What it breaks beyond the test
harness benchgetsTemplatesfrom this parser (internal/sitebench/bench.go:178, and:199with--hints). On real Hugo,--jsontemplatesis always[]and thecachePotentialcolumn never appears. The text output silently drops the "where to look" table (cmd/harness/bench.go:257).Why the unit tests did not catch it
The fixtures were hand-written in Go duration format, not taken from Hugo:
internal/sitebench/build_test.go:20and:31cmd/harness/bench_test.go:75and:84(the stub hugo)The PR body notes Hugo was not available locally, so the only real-Hugo run was in CI, and that is where it failed.
What to fix
16.64 s,7.52 ms,201.12 ms,63.28 µs) before parsing, or parse the counts and template from the end of the row as main did. Keep handling both table forms.internal/sitebench/build_test.goand the stub tables incmd/harness/bench_test.gowith a table captured from Hugo 0.167. The CI log of run 399 has a complete hints table. Add a plain--templateMetricstable too, so a format drift fails without Hugo.go test ./internal/themewith Hugo on PATH, or let CI do it, and confirmtemplateMetricspasses.Checked and fine
Closes #363is present.scaffold/skills_test.gochange follows the one-owner rule.docs/harness.mdare updated per CLAUDE.md.internal/sitebench,cmd/harness,docsandscaffold.Hugo 0.167 prints each duration in the --templateMetrics table as a number and a unit in separate fields ("16.64 s", "7.52 ms", "63.28 µs"), not as a Go duration string. parseRow ran time.ParseDuration on the bare number and rejected every row, so harness bench reported no templates and the guard test's once-per-key subtest found no footer-pages.html or news-items.html. parseDuration now joins a number with the unit field after it, and still reads a single Go duration string (Hugo 0.161's form). The parser tests read testdata/hugo-0.167-hints.txt, the table Hugo 0.167.0 printed in CI run 399, and a plain-layout table with the same rows. The stub hugo in cmd/harness prints those files, and the fakes in the sitebench tests print Hugo 0.167's format. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>[autoland] verdict: fix-done
[autoland] verdict: pass
Verdict: CONCERNS (approved, not merged)
Signal relied on:
labctl pr checks 367 --wait→success, from CI run 400 on head6b1f7fb. I read the job log (/Cloonar/deckle/actions/runs/400/jobs/0/attempt/1/logs). Hugo 0.167.0 extended was installed.go vet ./...andgo test ./...were green.internal/themeran for 50.3 s, soTestBuildTimeScalesLinearlyand itstemplateMetricssubtest ran against real Hugo rather than skipping.internal/sitebench,cmd/harness,docsandscaffoldall passed.The previous rejection is resolved
internal/sitebench/build.goparseDurationnow joins a number with the unit field that follows it.metrics/metrics.go:formatDurationprints%.2f s,%.2f ms,%.2f µsand%.2f ns. All four units are indurationUnits.%15s %12s %12s %5d %s(sum, avg, max, count, key).%9d %7.f %6d %5d.testdata/hugo-0.167-plain.txthas the same shape as real Hugo output. The hints fixture was captured from CI run 399.Concern (non-blocking): the real-Hugo acceptance criteria were never run
CI exercises real Hugo only through the guard test, which covers
Measure,RunBuildand the hints-form parser on the shipped theme. It does not cover the following, and the PR body says they are worth doing by hand before merging:harness bench --pages 500on a fresh site with news and categories. This exercisessitebench.Runend to end:copySite,resolveSection, andItemswritten into a real news section. The default plain-table path is also only checked against real Hugo by the source comparison above.--pages 2000→slowand exit 1; the fixed theme →ok. Unlike the guard, the bench passes--templateMetricson every timed build, so this confirms the 2.7 limit still separates the two with that overhead included.--hintson the fixed site showscachePotential, and--timeout 1sreproduces the hints timeout against old partials at 2,534 items.The last round showed that stub-only coverage can miss a real-Hugo-only bug. A maintainer should run these three by hand before merging. The lander could not, because downloading Hugo was not permitted in this run.
Checked and fine
Closes #363is present.resolveHugoextraction fromserveis what lets the bench use the pinned Hugo.scaffold/skills_test.gonow reads the command set fromdocs/harness.md, which removes a hand copy (the one-owner rule).docs/harness.md"Bench" and "harness bench --json" sections, the one README row, the AGENTS.md "Verifying a change" paragraph, one theme README "Navigation" sentence, and the docs-map entry.verify-sitehas the new step 3.redo-themehas the new step 4, and the later steps were renumbered; I checked that no step reference elsewhere is left stale.migrate-site(steps 7 and 9),create-featureandadd-page-typepoint at verify-site without restating the thresholds.harness checkis unchanged.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.