Theme: three partials scan every page on every page, making the build quadratic (gbv-aktuell: 2,573 pages) #360
Labels
No labels
bug
enhancement
needs-triage
ready-for-agent
ready-for-human
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Cloonar/deckle#360
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Three partials scan every page of the site on every page, making a build
quadratic in the page count (2,565 pages: 19 s on one core on the lab box,
28.6 s on the reporter's, about 2.5 min on a hosted vCPU). Compute each
site-wide selection once per build through a cached partial, skip the
children scan for sections whose dropdown cannot show, bump the news bundle,
document the pattern, have
harness checkwarn on the pattern, and add aguard test so it cannot come back.
Current behavior
partials/header.htmlcomputes every top-level section's children onevery page before checking
hideChildren; for a news section that is awhereover every item, per page.partials/footer.htmlrunswhere hugo.Sites.Default.Pages.ByWeight "Params.footer" trueon every page.partials/pagetypes/news.htmlre-filters and re-sortsevery item of the section on each pager page.
Reproduced during triage on a synthetic site (N news items under one
newssection, 12 categories, 13 top-level pages each placing a
news-list, thestock theme plus the news and categories bundles assembled the way
internal/theme'sbuildSiteWithThemedoes, Hugo 0.167.0 extended,GOMAXPROCS=1):Stock is superlinear (2.1x pages, 3.2x time); patched is near linear (2.2x).
hugo --templateMetrics --templateMetricsHintslistsheader.html,footer.html,term.html,pagetypes/news.htmlin that order. Hugo printedno WARN or ERROR on the patched build, and with the whitespace shape of
header.htmlkept the output was byte-identical (diff -r). Saves in thepreview pay the same cost (see Original report; not re-measured).
Desired behavior
can show: the entry is a section and its source page does not set
hideChildren: true. Menu output is unchanged for every page.inside a cached partial that holds the scan itself;
footer.htmlreachesit through
partialCachedkeyed by the language.to the current language, drafts out) is computed once per section per
language per build inside a cached partial;
pagetypes/news.htmlpaginates what it returns.
byte-identical (
diff -r), whitespace included: keep the whitespace shapeof
header.htmlrather than adding{{-trims (the diff in the Originalreport adds trims and drops one blank line per nav entry on every page).
versionbecomes 8.harness add newson an existingsite fills the new partial in; it never overwrites the existing
news.htmlandharness migraterewrites sidecars only, sodocs/features.mdsays what to apply by hand, as its version 5 note does.scaffold/site/hugo.tomlsets[pagination] pagerSize = 20with acomment saying why (it halves the pager pages every save re-renders). The
two documentation lines that say the pager size is "Hugo's default 10"
(
scaffold/site/AGENTS.md,docs/features.md) say what the scaffold sets.harness checkwarns on a template that reads a site-wide pagecollection (
hugo.Sites.Default.Pages,site.Pages,.Site.Pages,their
RegularPagesforms, andhugo.Sites.Default.RegularPages) unlessthe template is a partial that every caller reaches through
partialCached. The warning names the template and says to move the scaninto a partial reached through
partialCached. A section's own.Pages(
$src.Pages) is not warned on: its size cannot be judged statically. Afresh site, the scaffold theme and the shipped bundles produce no such
warning.
internal/themebuilds a generated site of N and 2N pages inthis shape with real
hugoon one core and fails when the build-timeratio shows the quadratic pattern. It skips without
hugoon PATH andunder
-short, like its siblings.and the feature docs point at it.
Decisions
partialCached "l10n/pages.html" (where …)evaluates thewherebeforethe cache lookup, so it saves nothing.
site.Language.Langand the news items by the sourcesection and the language: each selection is identical for every page of
a language (and section), and the mapped, draft-filtered result is per
language.
backendlayout/default.html,l10n/t-en.html: "constant per site, callers reach it throughpartialCached") so no second idiom appears.
change is provably a pure optimisation.
term.htmlandelements/news-list.htmlare not changed: caching theterm scan measured no gain during triage (its cost is drawing the cards,
7.39 s vs 7.48 s), and the element runs once per placement.
harness checkwarning ships with the fix rather than asa follow-up. Its rule, "which templates read a site-wide collection, and
is each reached only through
partialCached", is owned byinternal/tplrefs(it already owns the partial-call rulesharness checkuses and already recognises
partialCachedandpartials.IncludeCachedas call forms);checkcmdcalls it and the themetest suite calls the same function rather than grepping on its own.
docs/features.md"Not coveredyet"); the hand-apply note is the upgrade path.
redo-themeandmigrate-sitename the pattern a fork must follow, and
harness checkreports it.large site, on the order of the reporter's (at least 2,000 items at N),
accepting the test's running time; calibrate the threshold from the
measurement above (stock 3.2x, patched 2.2x at 1231→2565 pages) with
margin on both sides, and prove it fails against the stock partials
before pinning it.
Key interfaces
scaffold/themes/deckle/layouts/partials/header.html,footer.html, anew
footer-pages.htmlthat returns a page collection.scaffold/features/news/layouts/partials/pagetypes/news.html, a newnews-items.htmlwhose context is the source news section;scaffold/features/news/feature.yamlversion.scaffold/site/hugo.toml.internal/tplrefs(Calls,Templates,Provided): a rule thatclassifies each template's site-wide collection reads and whether its
callers all use
partialCached;internal/checkcmd(partialCallsisthe neighbouring check,
AreaTemplatesthe area).internal/theme:buildSiteWithThemeshows how the tests assemble asite with the theme and both bundles; the generator for the N-page site
is test code beside it.
scaffold/themes/deckle/README.md"Navigation" (the pattern) and "Layout files" (the new partial);
docs/harness.md"Check" (the new warning, in the warnings list and thePartials paragraph);
docs/features.md"The news bundle" (version,file count 19→20, upgrade note) and its pager-size line;
scaffold/site/AGENTS.md's pager-size line;scaffold/site/.claude/skills/redo-theme/references/theme-contract.md(a new numbered item under "Content model the theme reads");
redo-themestep 4 (navigation and footer) and itsharness checkparagraph,
migrate-sitestep 7 (porting the old theme),create-feature's section-page pattern,verify-sitewhere it readstemplateswarnings.Acceptance criteria
header.htmlevaluates a section's children only when.IsSectionand the source page's
hideChildrenis not true.footer.htmlandpagetypes/news.htmlreach their selections throughpartialCachedpartials that contain the scan; the template metrics ofa build show each cached partial executed once per key.
after, and Hugo prints no WARN or ERROR.
the fixed ones with margin; it skips without
hugoand under-short.version: 8;docs/features.mdlists the new partial, thenew file count and the hand-apply step for an older install.
scaffold/site/hugo.tomlsetspagerSize = 20; the two docs lines thatsaid "default 10" now say what the scaffold sets.
harness checkwarns on a site layout or partial that reads asite-wide collection outside a
partialCached-only partial (a testwith a fixture template), and reports nothing on a fresh site with the
news and categories bundles installed;
docs/harness.mddocuments thewarning and
go test ./cmd/harness/ ./docs/pass.and
create-featurepoint at it;go test ./scaffold/ ./docs/ ./internal/element/pass.How to verify
go test ./...withhugoon PATH (theinternal/themeintegrationtests, the skills and docs drift tests,
scaffold/canonical_test.go, thecmd/harnessusage test).diff -rof two production builds of the synthetic site.hugo --templateMetrics --templateMetricsHintson the synthetic site:footer-pages.htmlandnews-items.htmlshowcached≈total− keys.harness check --site <fresh site with news>prints no new warning; thesame site with the stock
footer.htmlcopied intolayouts/partials/prints one.
Assumptions to check
reporter's; the fix was measured on full builds only.
partialCachedon every server rebuild, so a new itemappears in the list on the next save.
hugo.tomlmay touchinitcmdfixtures orsnapshot tests.
tplrefs.Callsscans for literal partial names only; the new rule needsthe call form (cached or not) per call, which the scanner does not
return today.
Out of scope
term.htmlandelements/news-list.html.layouts-update mechanism for bundles.
.Pagesscans.Blocked by
None - can start immediately
Original report
What happens
On a site of 2,562 pages (2,534 news items, 12 categories, 13 top-level pages with news lists, stock theme, Hugo 0.167.0) the full build takes 28.6 s on one core and is superlinear: 628 pages 2.6 s, 1,228 pages 6.3 s, 2,562 pages 28.6 s. In the hosted instance of gbv-aktuell (1 throttled vCPU, about 3.5x slower than the lab box) this is the 2.5 minute "Hugo wird gestartet…" after every cold start.
hugo --templateMetrics --templateMetricsHintsnames the cost:header.html$childrenof every top-level section on every page BEFORE checkinghideChildren; for the news section that is awhereover 2,534 pages per pagefooter.htmlwhere hugo.Sites.Default.Pages.ByWeight "Params.footer" trueon every pagepagetypes/news.html(news bundle)Saves suffer from the same partials: Hugo's fast render treats a paginator as one visited unit, so once the editor has looked at
/news/in the preview, every save re-renders all 254 pager pages plus the home and the pages that carry anews-list. That is 5.8 s per save on one fast core, about 20 s on the hosted vCPU, during which Hugo starves the harness on the shared core. Editing the home page puts Hugo in the same state for the rest of the session.Fix
Three changes, measured on the same site, build output byte-identical (
diff -rof 3,141 files):pagination.pagerSize = 20Note the footer:
partialCached "l10n/pages.html" (where …)does NOT help, because thewhereargument is evaluated before the cached call. The scan has to live inside the cached partial. The same applies toterm.htmlin the categories bundle (337 term pages, 3.8 ms each) and toelements/news-list.html, which are smaller and can follow the same shape.Also in scope
versionbumps (new partialnews-items.html);harness migrateandharness addcarry it to existing sites.themes/deckle/README.mdlists the partials; addfooter-pages.htmlandnews-items.html.internal/theme) that builds a generated site of N and 2N pages and fails when the ratio of build times exceeds, say, 2.5, so the quadratic pattern cannot come back. The generator used for the measurement is a 150-line Go program; it can live ininternal/theme/testdataor the harness.redo-themeandmigrate-siteskills should name the three patterns as things a fork must not contain, andharness checkcould warn onwhere hugo.Sites.Default.Pagesorwhere $src.Pagesoutside apartialCachedpartial.pagination.pagerSize = 20in the scaffold'shugo.tomlfor news sections; it halves the pages every save re-renders.Evidence
Hosted: deckle-hosted #163 and #164 (the instance caps and the cold-start experience); ADR 0030 (issue #358) fixed the poll pile-up, this is what was underneath. Measurements on a synthetic site generated to gbv-aktuell's shape; metrics tables and the generator are available on request.
CI verification report (PR #364)
This report comes from CI run 388 (
ci / build-and-test, head21ce185, Hugo v0.167.0 extended). That head had a temporary test,TestVerify360, which built every site twice: once with the fixed partials and once with copies of the stockheader.html,footer.htmlandpagetypes/news.html. The test was written to always fail so CI would print the report. It has been reverted ind98b30e, so the PR now carries only the real change.What this shows for the acceptance criteria:
scaleMaxRatio = 2.7. The stock partials scale at 3.42x, 0.72 over it. This matches triage (2.2x vs 3.2x), so the limit stays at 2.7.internal/themewasTestVerify360.TestBuildTimeScalesLinearlypassed, and so did itstemplateMetricssubtest, which asserts thatfooter-pages.htmlandnews-items.htmleach execute once per key (total - cached == 1). Every other package passed too.