Cloonar/deckle
3
0
Fork 0
Code Issues 3 Pull requests Projects Releases Packages Wiki Activity Actions

Shortcode round-trip engine: .md ⇄ element tree with canonical serialization #2

New issue
Closed
opened 2026-07-11 12:58:40 +02:00 by dominik.polakovics · 2 comments
dominik.polakovics commented 2026-07-11 12:58:40 +02:00
Owner
Copy link

What to build

The highest-risk component, as a standalone Go package with a debug surface (docs/DECISIONS.md D5): parse a Hugo content file (front matter + body) into an element tree using Hugo's vendored parser/pageparser, and serialize the tree back to a canonical, human-readable .md (stable ordering, indentation, quoting). Nested block shortcodes and inline shortcodes inside body text must survive losslessly. Field model: flat scalar params + one body per element.

Acceptance criteria

  • Parser produces an element tree (typed nodes: block shortcode, inline shortcode, text run, front matter) from files containing nested paired shortcodes and markdown bodies
  • Serializer emits the canonical format; serialize(parse(canonical)) is byte-identical
  • parse(serialize(tree)) is structurally identical for arbitrary trees (property-based/fuzz round-trip tests)
  • Non-canonical but valid hand-written input parses correctly and re-emits canonically
  • Param values with quotes and special characters round-trip; newlines in params are rejected with a clear error
  • Debug endpoint on the tool returns the parsed tree of a given page as JSON
  • Test corpus covers every planned v1 element shape (grid nesting, accordion items, hero with inline buttons in body)

Blocked by

None - can start immediately

## What to build The highest-risk component, as a standalone Go package with a debug surface (docs/DECISIONS.md D5): parse a Hugo content file (front matter + body) into an element tree using Hugo's vendored `parser/pageparser`, and serialize the tree back to a canonical, human-readable `.md` (stable ordering, indentation, quoting). Nested block shortcodes and inline shortcodes inside body text must survive losslessly. Field model: flat scalar params + one body per element. ## Acceptance criteria - [ ] Parser produces an element tree (typed nodes: block shortcode, inline shortcode, text run, front matter) from files containing nested paired shortcodes and markdown bodies - [ ] Serializer emits the canonical format; serialize(parse(canonical)) is byte-identical - [ ] parse(serialize(tree)) is structurally identical for arbitrary trees (property-based/fuzz round-trip tests) - [ ] Non-canonical but valid hand-written input parses correctly and re-emits canonically - [ ] Param values with quotes and special characters round-trip; newlines in params are rejected with a clear error - [ ] Debug endpoint on the tool returns the parsed tree of a given page as JSON - [ ] Test corpus covers every planned v1 element shape (grid nesting, accordion items, hero with inline buttons in body) ## Blocked by None - can start immediately
dominik.polakovics commented 2026-07-11 14:20:11 +02:00
Author
Owner
Copy link

This was generated by AI during triage.

Agent Brief

Category: enhancement
Summary: Build the shortcode round-trip engine: a standalone Go package that parses Hugo content files (front matter + body with nested and inline shortcodes) into a typed element tree and serializes the tree back to a canonical, human-readable .md, plus a debug endpoint that exposes the parsed tree as JSON.

Note on references: the issue body cites docs/DECISIONS.md D5; that file is not committed to the repository. The relevant decision content is restated inline here — this brief is self-contained.

Current behavior:
The repository contains the walking skeleton from #1: a harness serve --site <dir> binary that supervises hugo server and serves an embedded SolidJS shell with an HTTP API under /api. There is no content parsing of any kind — the tool never reads .md files. The sample site's content files are plain front-matter posts with no shortcodes, so no test corpus exists yet.

Desired behavior:
A standalone Go package (no dependencies on the server or supervisor packages; those may depend on it) that implements a lossless, canonicalizing round-trip between Hugo content files and an element tree:

  • Parsing. Given the bytes of a Hugo content file, produce a tree of typed nodes: front matter, block (paired) shortcode, inline (self-closing) shortcode, and text run. Use Hugo's parser/pageparser package (importable as github.com/gohugoio/hugo/parser/pageparser) as the lexer rather than hand-rolling shortcode tokenization. Nested paired shortcodes (e.g. a grid containing columns containing further elements) become nested tree nodes; inline shortcodes interleaved with markdown text inside a body (e.g. a button inside a hero's text) survive as child nodes positioned within the text runs.
  • Field model. Each element (shortcode node) carries flat scalar named params only — no nested or array param values — plus at most one body (its inner content). Param values containing quotes and special characters must round-trip exactly. Param values containing newlines are rejected at serialization time with a clear, actionable error naming the offending element and param.
  • Canonical serialization. The serializer emits one canonical form: stable param ordering, stable indentation for nested block shortcodes, and consistent quoting rules. Canonical output is human-readable and diff-friendly — a human editing the file in git should recognize it.
  • Round-trip guarantees. serialize(parse(x)) is byte-identical when x is already canonical. parse(serialize(tree)) is structurally identical to tree for arbitrary valid trees. Non-canonical but valid hand-written input (different quoting, param order, whitespace) parses correctly and re-emits in canonical form.
  • Debug surface. The harness HTTP API gains a debug endpoint that, given a page identifier (e.g. a content-relative path), returns the parsed element tree of that page as JSON. This is a developer/diagnostic surface, not a UI feature — no frontend work is needed beyond what already exists.

Key interfaces:

  • The package's public API should expose at minimum: a parse function (file bytes → tree, with errors carrying position info), a serialize function (tree → file bytes), and exported node types distinguishing front matter, block shortcode, inline shortcode, and text runs. Exact naming is the implementer's choice.
  • Tree nodes must be JSON-serializable (the debug endpoint returns them) with a stable, typed shape — a discriminator field for node kind, params as an ordered map or sorted list, children for nested content.
  • Front matter is preserved through the round-trip. Parsing its internal structure into typed fields is NOT required here — treating it as an opaque preserved block is acceptable; field-level editing arrives with #3.
  • The debug endpoint lives on the existing harness HTTP server alongside the current /api routes.

Acceptance criteria:

  • Parser produces an element tree with typed nodes (block shortcode, inline shortcode, text run, front matter) from files containing nested paired shortcodes and markdown bodies
  • Serializer emits the canonical format; serialize(parse(x)) is byte-identical for canonical x
  • parse(serialize(tree)) is structurally identical for arbitrary trees, demonstrated by property-based/fuzz round-trip tests (Go's native fuzzing or a generator-driven property test)
  • Non-canonical but valid hand-written input parses correctly and re-emits canonically
  • Param values with quotes and special characters round-trip exactly; newlines in param values are rejected with a clear error identifying the element and param
  • A debug endpoint on the harness returns the parsed tree of a given page as JSON
  • A test corpus of content files covers every planned v1 element shape — at minimum: grid with nested columns and nested elements inside them, accordion with item children, hero with inline button shortcodes inside its body text — and lives in the repo where tests consume it
  • The package builds and tests pass in CI alongside the existing suite

Out of scope:

  • Sidecar schemas, generated forms, and field-level front-matter editing (#3)
  • Element CRUD operations on the tree — add/remove/reorder/nest (#6)
  • Rich-text editing of bodies and shortcode chips in the editor (#7, #8)
  • Defining the actual v1 element library as Hugo shortcode templates (#14) — this issue only needs representative corpus files exercising the shapes, not working templates that render
  • Any UI for viewing the tree — the debug endpoint returning JSON is sufficient
  • Git operations, saving through the API as a user-facing feature (#10)
> *This was generated by AI during triage.* ## Agent Brief **Category:** enhancement **Summary:** Build the shortcode round-trip engine: a standalone Go package that parses Hugo content files (front matter + body with nested and inline shortcodes) into a typed element tree and serializes the tree back to a canonical, human-readable `.md`, plus a debug endpoint that exposes the parsed tree as JSON. **Note on references:** the issue body cites `docs/DECISIONS.md D5`; that file is not committed to the repository. The relevant decision content is restated inline here — this brief is self-contained. **Current behavior:** The repository contains the walking skeleton from #1: a `harness serve --site <dir>` binary that supervises `hugo server` and serves an embedded SolidJS shell with an HTTP API under `/api`. There is no content parsing of any kind — the tool never reads `.md` files. The sample site's content files are plain front-matter posts with no shortcodes, so no test corpus exists yet. **Desired behavior:** A standalone Go package (no dependencies on the server or supervisor packages; those may depend on it) that implements a lossless, canonicalizing round-trip between Hugo content files and an element tree: - **Parsing.** Given the bytes of a Hugo content file, produce a tree of typed nodes: front matter, block (paired) shortcode, inline (self-closing) shortcode, and text run. Use Hugo's `parser/pageparser` package (importable as `github.com/gohugoio/hugo/parser/pageparser`) as the lexer rather than hand-rolling shortcode tokenization. Nested paired shortcodes (e.g. a grid containing columns containing further elements) become nested tree nodes; inline shortcodes interleaved with markdown text inside a body (e.g. a button inside a hero's text) survive as child nodes positioned within the text runs. - **Field model.** Each element (shortcode node) carries flat scalar named params only — no nested or array param values — plus at most one body (its inner content). Param values containing quotes and special characters must round-trip exactly. Param values containing newlines are rejected at serialization time with a clear, actionable error naming the offending element and param. - **Canonical serialization.** The serializer emits one canonical form: stable param ordering, stable indentation for nested block shortcodes, and consistent quoting rules. Canonical output is human-readable and diff-friendly — a human editing the file in git should recognize it. - **Round-trip guarantees.** `serialize(parse(x))` is byte-identical when `x` is already canonical. `parse(serialize(tree))` is structurally identical to `tree` for arbitrary valid trees. Non-canonical but valid hand-written input (different quoting, param order, whitespace) parses correctly and re-emits in canonical form. - **Debug surface.** The harness HTTP API gains a debug endpoint that, given a page identifier (e.g. a content-relative path), returns the parsed element tree of that page as JSON. This is a developer/diagnostic surface, not a UI feature — no frontend work is needed beyond what already exists. **Key interfaces:** - The package's public API should expose at minimum: a parse function (file bytes → tree, with errors carrying position info), a serialize function (tree → file bytes), and exported node types distinguishing front matter, block shortcode, inline shortcode, and text runs. Exact naming is the implementer's choice. - Tree nodes must be JSON-serializable (the debug endpoint returns them) with a stable, typed shape — a discriminator field for node kind, params as an ordered map or sorted list, children for nested content. - Front matter is preserved through the round-trip. Parsing its internal structure into typed fields is NOT required here — treating it as an opaque preserved block is acceptable; field-level editing arrives with #3. - The debug endpoint lives on the existing harness HTTP server alongside the current `/api` routes. **Acceptance criteria:** - [ ] Parser produces an element tree with typed nodes (block shortcode, inline shortcode, text run, front matter) from files containing nested paired shortcodes and markdown bodies - [ ] Serializer emits the canonical format; `serialize(parse(x))` is byte-identical for canonical `x` - [ ] `parse(serialize(tree))` is structurally identical for arbitrary trees, demonstrated by property-based/fuzz round-trip tests (Go's native fuzzing or a generator-driven property test) - [ ] Non-canonical but valid hand-written input parses correctly and re-emits canonically - [ ] Param values with quotes and special characters round-trip exactly; newlines in param values are rejected with a clear error identifying the element and param - [ ] A debug endpoint on the harness returns the parsed tree of a given page as JSON - [ ] A test corpus of content files covers every planned v1 element shape — at minimum: grid with nested columns and nested elements inside them, accordion with item children, hero with inline button shortcodes inside its body text — and lives in the repo where tests consume it - [ ] The package builds and tests pass in CI alongside the existing suite **Out of scope:** - Sidecar schemas, generated forms, and field-level front-matter editing (#3) - Element CRUD operations on the tree — add/remove/reorder/nest (#6) - Rich-text editing of bodies and shortcode chips in the editor (#7, #8) - Defining the actual v1 element library as Hugo shortcode templates (#14) — this issue only needs representative corpus files exercising the shapes, not working templates that render - Any UI for viewing the tree — the debug endpoint returning JSON is sufficient - Git operations, saving through the API as a user-facing feature (#10)
dominik.polakovics commented 2026-07-12 02:01:39 +02:00
Author
Owner
Copy link

This was generated by AI while landing a PR.

Landing audit — PR #19 (afk/2)

Verdict: PASS (with two minor, non-blocking notes on the debug endpoint).

Checked:

  • CI: green — ci / build-and-test (pull_request) success (2m31s). Relied on as the vouching signal; not re-run.
  • Convention: title is Conventional Commits (feat:). AFK contract holds — head afk/2, body carries Closes #2.
  • Mergeability: merges into main cleanly (no conflicts).
  • Acceptance criteria (issue #2): all met — typed element tree; byte-identical serialize(parse(x)); structural parse(serialize(tree)) (property + native fuzz); non-canonical→canonical re-emit; quote/special-char params round-trip and newlines rejected; /api/debug/tree JSON endpoint; v1-shape corpus (grid/accordion/hero-inline).
  • Adversarial diff review: an independent pass (75s / 5.2M-exec fuzz + manual probes of JSON/Org front matter, comment-escapes, summary dividers, BOM, no-FM leading bytes, nested same-name blocks) found no correctness bugs and no silent round-trip breaks. Path traversal on /api/debug/tree is safe (filepath.Clean + IsLocal + Join; every escape attempt rejected 400).

Minor non-blocking notes (debug/diagnostic endpoint only, internal/server/server.go):

  1. A lexically-local page that resolves to a directory (e.g. page=.) yields EISDIR → 500 rather than a 4xx.
  2. That 500 branch interpolates err.Error(), leaking the absolute server path. Consider stat-and-reject with 404 and not echoing the raw error.

Neither affects the round-trip guarantees or acceptance criteria; safe to land as-is and tighten later if desired.

> *This was generated by AI while landing a PR.* **Landing audit — PR #19 (`afk/2`)** **Verdict: PASS** (with two minor, non-blocking notes on the debug endpoint). Checked: - **CI**: green — `ci / build-and-test (pull_request)` success (2m31s). Relied on as the vouching signal; not re-run. - **Convention**: title is Conventional Commits (`feat:`). AFK contract holds — head `afk/2`, body carries `Closes #2`. - **Mergeability**: merges into `main` cleanly (no conflicts). - **Acceptance criteria** (issue #2): all met — typed element tree; byte-identical `serialize(parse(x))`; structural `parse(serialize(tree))` (property + native fuzz); non-canonical→canonical re-emit; quote/special-char params round-trip and newlines rejected; `/api/debug/tree` JSON endpoint; v1-shape corpus (grid/accordion/hero-inline). - **Adversarial diff review**: an independent pass (75s / 5.2M-exec fuzz + manual probes of JSON/Org front matter, comment-escapes, summary dividers, BOM, no-FM leading bytes, nested same-name blocks) found **no correctness bugs** and no silent round-trip breaks. Path traversal on `/api/debug/tree` is safe (`filepath.Clean` + `IsLocal` + `Join`; every escape attempt rejected 400). Minor non-blocking notes (debug/diagnostic endpoint only, `internal/server/server.go`): 1. A lexically-local `page` that resolves to a **directory** (e.g. `page=.`) yields `EISDIR` → 500 rather than a 4xx. 2. That 500 branch interpolates `err.Error()`, leaking the absolute server path. Consider stat-and-reject with 404 and not echoing the raw error. Neither affects the round-trip guarantees or acceptance criteria; safe to land as-is and tighten later if desired.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
afk/130
afk/129
afk/128
afk/124
afk/117
afk/116
afk/115
afk/114
afk/113
afk/112
afk/111
lab/20260718-0231
afk/95
afk/93
afk/91
afk/97
afk/90
afk/94
afk/96
afk/92
afk/89
lab/20260716-2349
afk/83
afk/72
afk/70
afk/58
afk/56
afk/54
afk/52
afk/50
afk/48
lab/20260715-1223
afk/44
afk/42
afk/40
afk/37
afk/35
afk/33
lab/20260712-0208
afk/9
afk/4
afk/14
afk/12
afk/10
afk/7
afk/6
afk/3
afk/2
afk/1
No results found.
Labels
Clear labels   
enhancement
New feature or improvement

  
needs-triage
Maintainer needs to evaluate this issue

  
ready-for-agent
Fully specified, ready for an AFK agent

  
ready-for-human
Requires human implementation

No labels
enhancement
needs-triage
ready-for-agent
ready-for-human
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Cloonar/deckle#2
No description provided.