Cloonar/nixos
3
0
Fork 0
Code Issues 31 Pull requests 7 Projects Releases Packages Wiki Activity Actions

feat(web-arm): reptide crash-signal prerequisites — 90d stream retention, push-only Loki vhost, error-rate alert #325

Merged
dominik.polakovics merged 3 commits from afk/323 into main 2026-08-21 21:15:16 +02:00
Conversation 1 Commits 3 Files changed 4 +113 -4
dominik.polakovics commented 2026-08-10 23:33:38 +02:00
Owner
Copy link

Prerequisites from this repo for reptide's crash-signal pipeline (Flutter → Supabase log-event → Loki). Two commits, per the issue.

1. hosts/web-arm/modules/loki.nix

90d retention for reptide, 48h for everyone else — a per-stream override on {app="reptide"} (2160h, priority 1) rather than a change to the global period. The compactor already had retention_enabled = true.

The public vhost becomes push-only. Both fleet pushers (utils/modules/alloy/config.alloy, ansible/roles/base/defaults/main.yml) and reptide's edge function only ever POST to /loki/api/v1/push; Grafana queries Loki over localhost. So the vhost now exposes an exact-match push location behind the existing basic auth, an open /ready, and denies everything else with 403. That closes the read hole for any leaked pusher credential — the fleet's or reptide's.

Certificate renewal is unaffected: security.acme.defaults.dnsProvider = "hetzner" (DNS-01), so no HTTP-01 webroot path passes through the vhost.

2. hosts/web-arm/modules/grafana/alerting/reptide/

Error-rate alert per MANUAL_SETUP.md §8: 5m error rate for {app="reptide", env="production", level="error"} above 3× the same window 24h earlier, sustained 10m, routed by the existing default policy to cp_dominik_normal (the Matrix room in the doc is reptide-side wiring, not available here). No dashboard — Explore covers early triage until a real incident shows what is worth freezing into Nix.

One deliberate deviation from the literal spec, worth a look: both queries end in or vector(0). Without it, a silent 24h baseline returns an empty vector, the math step degrades to NoData, and the rule stays mute in exactly the case that matters — a healthy app that suddenly starts crashing. The trade-off is that a zero baseline makes the threshold zero, so after a quiet 24h any error sustained over 10m fires. Given this is a crash signal that seems right, but it is a real behaviour change from "3× baseline" and easy to revert if it turns out noisy.

Verification

Nix is not installed in the AFK sandbox, so I bootstrapped a portable Nix (third-party binary, DavHau/nix-portable v012, into $HOME outside the repo) to get the repo's build gate working. Everything below was run against packages from the host's own channel, nixos-26.05.

Dry-build: scripts/test-configuration web-arm via the pre-commit hook → :: web-arm OK, zero evaluation errors.

Retention, against a real Loki 3.7.4 — the rendered config was started locally, and its own GET /config reads back:

limits_config.retention_period: 2d
limits_config.retention_stream:
  - period: 90d
    priority: 1
    selector: '{app="reptide"}'
compactor.retention_enabled: true

2160h → 90d and 48h → 2d are Go duration round-trips, so the values were parsed, not echoed. Three falsification controls confirm the parser is strict rather than tolerant: retention_streamm → field not found in type validation.plain; selectorr → field not found in type validation.StreamRetention; malformed selector {app="reptide" → invalid labels matchers: parse error at line 1, col 15. All exit 1. Loki LogQL-parses the selector at config-validation time.

LogQL, same instance: with no data, sum(rate(...)) or vector(0) returns a sample of 0 where the bare query returns [] — the behaviour the alert depends on. After pushing 5 level="error" lines, query A = 0.01666… (exactly 5/300); 3 additional level="info" lines did not move it (they landed — level="info" alone = 0.01, no level matcher = 0.02666…). With A = 0.01666… and B = 0, $C > 3 * $D is true; on total silence 0 > 3*0 is false, so or vector(0) does not manufacture a false positive.

nginx, real nginx 1.30.4 from the same channel, serving the three rendered location blocks verbatim against a stub upstream — all 11 rows pass:

request result
POST /loki/api/v1/push no creds 401
POST /loki/api/v1/push valid creds 200, reached upstream
GET /ready no creds 200, reached upstream
GET /loki/api/v1/query_range no creds / valid creds 403 / 403
GET /loki/api/v1/labels valid creds 403
GET /config, GET /metrics (± creds) 403
GET / 403
POST /loki/api/v1/push/ (trailing slash) valid creds 403
POST /loki/api/v1/push?x=1 valid creds 200
GET /ready/ (trailing slash) 403

The stub upstream's own log confirms only the authenticated push, /ready, and the query-string push ever reached it. Note row 5: valid credentials do not buy query access — that is the point of the change.

Grafana provisioning: the rendered rules.yaml contains all five groups (System 5, Service 11, Website 45, Storage 8, Reptide 1 — the existing four are intact), 70 rules with 70 unique UIDs, and reptide-error-rate-spike-uid appears exactly once, so alert-cleanup.nix will not reap it.

/ready returning 503: does not reproduce. https://loki.cloonar.com/ready returned 200 ready on 6 consecutive probes before the change. Loki serves 503 there only while the ingester is warming up, which fits a transient observation shortly after a deploy. loki.cloonar.com should stay on the blackbox blacklist regardless — after this change GET / returns 403, which a plain http_200 probe would read as down.

Not verified here

  • Post-deploy checks (unauthenticated surface on the live host, fleet journals still flowing, Coredumps ruler alert intact) can only run after this merges and bento pulls.
  • Retention enforcement — I proved the config is accepted and reads back correctly, not that Loki actually drops fleet data at 48h and keeps reptide for 90d; that needs data older than 48h.
  • Grafana did not evaluate the rule. The alert JSON is proven to render and provision; that Grafana's Loki datasource accepts queryType: instant with these expressions is verified only at the LogQL level.

Operator steps (not doable from here — secrets are yours to edit)

  1. Add a reptide-prod htpasswd line to the existing promtail-nginx-password secret (it already holds the fleet's promtail@cloonar.com line; nginx reads multiple lines from one file): 
    nix-shell -p sops --run 'sops hosts/web-arm/secrets.sops.yaml'
    No new secret and no deploy ordering constraint — this can land before or after the merge.
  2. Same password into Supabase per MANUAL_SETUP.md §3: supabase secrets set LOKI_PUSH_URL="https://loki.cloonar.com/loki/api/v1/push" LOKI_USERNAME="reptide-prod" LOKI_PASSWORD=… LOKI_ENV="production", then deploy the log-event edge function.
  3. Confirm an authenticated push returns 204 and {app="reptide"} shows up in Grafana Explore.

Closes #323

Prerequisites from this repo for reptide's crash-signal pipeline (Flutter → Supabase `log-event` → Loki). Two commits, per the issue. ## 1. `hosts/web-arm/modules/loki.nix` **90d retention for reptide, 48h for everyone else** — a per-stream override on `{app="reptide"}` (2160h, priority 1) rather than a change to the global period. The compactor already had `retention_enabled = true`. **The public vhost becomes push-only.** Both fleet pushers (`utils/modules/alloy/config.alloy`, `ansible/roles/base/defaults/main.yml`) and reptide's edge function only ever POST to `/loki/api/v1/push`; Grafana queries Loki over localhost. So the vhost now exposes an exact-match push location behind the existing basic auth, an open `/ready`, and denies everything else with 403. That closes the read hole for any leaked pusher credential — the fleet's or reptide's. Certificate renewal is unaffected: `security.acme.defaults.dnsProvider = "hetzner"` (DNS-01), so no HTTP-01 webroot path passes through the vhost. ## 2. `hosts/web-arm/modules/grafana/alerting/reptide/` Error-rate alert per MANUAL_SETUP.md §8: 5m error rate for `{app="reptide", env="production", level="error"}` above 3× the same window 24h earlier, sustained 10m, routed by the existing default policy to `cp_dominik_normal` (the Matrix room in the doc is reptide-side wiring, not available here). No dashboard — Explore covers early triage until a real incident shows what is worth freezing into Nix. **One deliberate deviation from the literal spec, worth a look:** both queries end in `or vector(0)`. Without it, a silent 24h baseline returns an empty vector, the math step degrades to NoData, and the rule stays mute in exactly the case that matters — a healthy app that suddenly starts crashing. The trade-off is that a zero baseline makes the threshold zero, so after a quiet 24h *any* error sustained over 10m fires. Given this is a crash signal that seems right, but it is a real behaviour change from "3× baseline" and easy to revert if it turns out noisy. ## Verification Nix is not installed in the AFK sandbox, so I bootstrapped a portable Nix (third-party binary, `DavHau/nix-portable` v012, into `$HOME` outside the repo) to get the repo's build gate working. Everything below was run against packages from the host's own channel, `nixos-26.05`. **Dry-build:** `scripts/test-configuration web-arm` via the pre-commit hook → `:: web-arm OK`, zero evaluation errors. **Retention, against a real Loki 3.7.4** — the rendered config was started locally, and its own `GET /config` reads back: ```yaml limits_config.retention_period: 2d limits_config.retention_stream: - period: 90d priority: 1 selector: '{app="reptide"}' compactor.retention_enabled: true ``` `2160h → 90d` and `48h → 2d` are Go duration round-trips, so the values were parsed, not echoed. Three falsification controls confirm the parser is strict rather than tolerant: `retention_streamm` → `field not found in type validation.plain`; `selectorr` → `field not found in type validation.StreamRetention`; malformed selector `{app="reptide"` → `invalid labels matchers: parse error at line 1, col 15`. All exit 1. Loki LogQL-parses the selector at config-validation time. **LogQL, same instance:** with no data, `sum(rate(...)) or vector(0)` returns a sample of `0` where the bare query returns `[]` — the behaviour the alert depends on. After pushing 5 `level="error"` lines, query A = `0.01666…` (exactly 5/300); 3 additional `level="info"` lines did not move it (they landed — `level="info"` alone = 0.01, no level matcher = 0.02666…). With A = 0.01666… and B = 0, `$C > 3 * $D` is true; on total silence `0 > 3*0` is false, so `or vector(0)` does not manufacture a false positive. **nginx, real nginx 1.30.4 from the same channel**, serving the three rendered location blocks verbatim against a stub upstream — all 11 rows pass: | request | result | |---|---| | `POST /loki/api/v1/push` no creds | 401 | | `POST /loki/api/v1/push` valid creds | 200, reached upstream | | `GET /ready` no creds | 200, reached upstream | | `GET /loki/api/v1/query_range` no creds / valid creds | 403 / 403 | | `GET /loki/api/v1/labels` valid creds | 403 | | `GET /config`, `GET /metrics` (± creds) | 403 | | `GET /` | 403 | | `POST /loki/api/v1/push/` (trailing slash) valid creds | 403 | | `POST /loki/api/v1/push?x=1` valid creds | 200 | | `GET /ready/` (trailing slash) | 403 | The stub upstream's own log confirms only the authenticated push, `/ready`, and the query-string push ever reached it. Note row 5: valid credentials do not buy query access — that is the point of the change. **Grafana provisioning:** the rendered `rules.yaml` contains all five groups (System 5, Service 11, Website 45, Storage 8, Reptide 1 — the existing four are intact), 70 rules with 70 unique UIDs, and `reptide-error-rate-spike-uid` appears exactly once, so `alert-cleanup.nix` will not reap it. **`/ready` returning 503:** does not reproduce. `https://loki.cloonar.com/ready` returned `200 ready` on 6 consecutive probes before the change. Loki serves 503 there only while the ingester is warming up, which fits a transient observation shortly after a deploy. `loki.cloonar.com` should stay on the blackbox blacklist regardless — after this change `GET /` returns 403, which a plain `http_200` probe would read as down. ### Not verified here - **Post-deploy checks** (unauthenticated surface on the live host, fleet journals still flowing, `Coredumps` ruler alert intact) can only run after this merges and bento pulls. - **Retention enforcement** — I proved the config is accepted and reads back correctly, not that Loki actually drops fleet data at 48h and keeps reptide for 90d; that needs data older than 48h. - **Grafana did not evaluate the rule.** The alert JSON is proven to render and provision; that Grafana's Loki datasource accepts `queryType: instant` with these expressions is verified only at the LogQL level. ## Operator steps (not doable from here — secrets are yours to edit) 1. Add a `reptide-prod` htpasswd line to the existing `promtail-nginx-password` secret (it already holds the fleet's `promtail@cloonar.com` line; nginx reads multiple lines from one file): ``` nix-shell -p sops --run 'sops hosts/web-arm/secrets.sops.yaml' ``` No new secret and no deploy ordering constraint — this can land before or after the merge. 2. Same password into Supabase per MANUAL_SETUP.md §3: `supabase secrets set LOKI_PUSH_URL="https://loki.cloonar.com/loki/api/v1/push" LOKI_USERNAME="reptide-prod" LOKI_PASSWORD=… LOKI_ENV="production"`, then deploy the `log-event` edge function. 3. Confirm an authenticated push returns 204 and `{app="reptide"}` shows up in Grafana Explore. Closes #323
Reptide's crash events need 90d (the app's privacy doc allows <= 90d); fleet
journals stay at 48h, so the retention is a per-stream override on
{app="reptide"} rather than a change to the global period.

Every automated consumer of loki.cloonar.com only ever pushes (fleet Alloy,
ansible hosts, reptide's Supabase edge function) and Grafana queries over
localhost, so the public vhost no longer needs to expose the query API. It now
serves an exact-match push endpoint behind the existing basic auth plus an open
/ready, and denies everything else -- which closes the read hole for any leaked
pusher credential, reptide's or the fleet's.

Refs #323
Fires when the 5m error rate for {app="reptide", env="production"} exceeds 3x
the same window 24h earlier, sustained for 10m, and routes through the default
policy to cp_dominik_normal.

Both Loki queries end in `or vector(0)`: without it a silent baseline returns an
empty vector, the math step degrades to NoData and the rule stays mute in
exactly the case that matters -- a healthy app that suddenly starts crashing.
The trade-off is that a zero baseline makes the threshold zero, so after a quiet
24h any error sustained over 10m fires.

No dashboard: Explore covers early triage until a real incident shows what is
worth freezing into Nix.

Refs #323
dominik.polakovics commented 2026-08-10 23:49:29 +02:00
Author
Owner
Copy link

[autoland] verdict: pass

This was generated by AI while landing a PR.

CONCERNS (non-blocking) - landable. No PR checks exist (aggregate: none), so I ran the repo gate myself: scripts/test-configuration web-arm - dry-build OK. Verified against the rendered config, not the PR body: nginx emits exactly '= /loki/api/v1/push' (basic auth), '= /ready', and 'location / { return 403; }' - exact matches win over the prefix catch-all; DNS-01 ACME (lego.nix:11) so renewal is unaffected. Grafana renders all 5 groups, 70 rules, 70 unique UIDs, reptide-error-rate-spike-uid once, and alert-cleanup.nix derives its manifest from the same list so it will not reap it. Both repo pushers (alloy config.alloy:114, ansible main.yml:76) POST to exactly that path; the domain is already blackbox-blacklisted. Title and 'Closes #323' conform; diff is confined to the issue's four files.

CONCERN: loki.nix:77-80 (pre-existing, untouched) still sets table_manager.retention_deletes_enabled with retention_period 48h. Table-manager retention is table-granular and ignores per-stream overrides, so if it is live it would cap reptide at 48h and defeat the 90d goal. Very likely inert (table-manager is not in Loki 3.x's 'all' target), but the PR correctly notes retention enforcement was never proven - worth confirming once data crosses 48h. Not a reason to hold the merge.

Noted, not blocking: the 'or vector(0)' deviation degenerates the threshold to '> 0' after a quiet 24h (author flagged it, one-line revert); the annotation says '3x the same window 24h ago' but query B is a 24h-averaged baseline - the code matches the issue spec, only the prose is loose; the push location drops proxyWebsockets, so nginx now talks HTTP/1.0 upstream, harmless for buffered POSTs.

Operator steps (reptide-prod htpasswd line, Supabase secrets) remain outstanding and are not merge-blocking per the issue.

[autoland] verdict: pass > *This was generated by AI while landing a PR.* CONCERNS (non-blocking) - landable. No PR checks exist (aggregate: none), so I ran the repo gate myself: scripts/test-configuration web-arm - dry-build OK. Verified against the rendered config, not the PR body: nginx emits exactly '= /loki/api/v1/push' (basic auth), '= /ready', and 'location / { return 403; }' - exact matches win over the prefix catch-all; DNS-01 ACME (lego.nix:11) so renewal is unaffected. Grafana renders all 5 groups, 70 rules, 70 unique UIDs, reptide-error-rate-spike-uid once, and alert-cleanup.nix derives its manifest from the same list so it will not reap it. Both repo pushers (alloy config.alloy:114, ansible main.yml:76) POST to exactly that path; the domain is already blackbox-blacklisted. Title and 'Closes #323' conform; diff is confined to the issue's four files. CONCERN: loki.nix:77-80 (pre-existing, untouched) still sets table_manager.retention_deletes_enabled with retention_period 48h. Table-manager retention is table-granular and ignores per-stream overrides, so if it is live it would cap reptide at 48h and defeat the 90d goal. Very likely inert (table-manager is not in Loki 3.x's 'all' target), but the PR correctly notes retention enforcement was never proven - worth confirming once data crosses 48h. Not a reason to hold the merge. Noted, not blocking: the 'or vector(0)' deviation degenerates the threshold to '> 0' after a quiet 24h (author flagged it, one-line revert); the annotation says '3x the same window 24h ago' but query B is a 24h-averaged baseline - the code matches the issue spec, only the prose is loose; the push location drops proxyWebsockets, so nginx now talks HTTP/1.0 upstream, harmless for buffered POSTs. Operator steps (reptide-prod htpasswd line, Supabase secrets) remain outstanding and are not merge-blocking per the issue.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

   
enhancement

   
in-progress

   
needs-info

   
needs-triage

   
p0

   
ready-for-agent

   
ready-for-human

   
wontfix
No labels
bug
enhancement
in-progress
needs-info
needs-triage
p0
ready-for-agent
ready-for-human
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Cloonar/nixos!325
No description provided.