reptide crash-signal prerequisites: Loki retention override, push-only credential, Grafana provisioning #323

New issue

Closed

opened 2026-08-10 22:47:31 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-08-10 22:47:31 +02:00

Owner

Reptide's crash-reporting pipeline (Flutter → Supabase log-event edge function → Loki, see reptide MANUAL_SETUP.md) needs prerequisites from this repo before the operator-side setup can run. Loki/Grafana on web-arm already cover the rest (public push endpoint at https://loki.cloonar.com/loki/api/v1/push behind nginx basic auth, provisioned Loki datasource in Grafana).

Design was grilled; decisions final:

Multi-tenancy: skipped. Single reader; isolation achieved by locking the public endpoint down instead.
Retention: 90d for reptide events (privacy doc allows ≤90d), fleet journals stay at 48h.
Public vhost becomes push-only. Every automated consumer (fleet Alloy, ansible hosts) only pushes; Grafana queries via localhost; nothing uses the public query API. Lock loki.cloonar.com to location = /loki/api/v1/push (basic auth, existing htpasswd) + /ready (open); everything else denied. Closes the read-hole for any leaked pusher credential, reptide's or the fleet's.
No new secret. The reptide-prod htpasswd line is added to the existing promtail-nginx-password secret in hosts/web-arm/secrets.sops.yaml (operator edit via sops; same password goes to Supabase as LOKI_PASSWORD per MANUAL_SETUP.md §2–3). Deploy order independent of the secret edit.
Grafana: alert only, no dashboard. Explore covers early triage; a dashboard can be frozen into Nix after the first real incident shows what's useful.

Work (single PR, two commits)

hosts/web-arm/modules/loki.nix
- limits_config.retention_stream = [{ selector = ''{app="reptide"}''; priority = 1; period = "2160h"; }] (compactor already has retention_enabled = true)
- nginx vhost: exact-match push location with existing auth, /ready open, deny the rest
hosts/web-arm/modules/grafana/alerting/reptide/ — error-rate alert per MANUAL_SETUP.md §8: sum(rate({app="reptide", env="production", level="error"}[5m])) > 3 * <24h baseline offset 24h>, sustained 10m, routed via existing default policy → cp_dominik_normal (Pushover; the Matrix room in the doc is reptide-side wiring, not available here).

Verification

agent, post-deploy: unauthenticated surface (push → 401, query paths → 403/404, /ready → 200); fleet journals still flowing; Coredumps ruler alert intact; retention override in rendered config; look into /ready currently returning 503 (unmonitored — blackbox blacklists the domain)
operator: add reptide-prod htpasswd line, then authenticated push → 204; supabase secrets set + edge-function deploy per MANUAL_SETUP.md; {app="reptide"} visible in Grafana Explore

Reptide's crash-reporting pipeline (Flutter → Supabase `log-event` edge function → Loki, see reptide `MANUAL_SETUP.md`) needs prerequisites from this repo before the operator-side setup can run. Loki/Grafana on web-arm already cover the rest (public push endpoint at `https://loki.cloonar.com/loki/api/v1/push` behind nginx basic auth, provisioned Loki datasource in Grafana). Design was grilled; decisions final: - **Multi-tenancy: skipped.** Single reader; isolation achieved by locking the public endpoint down instead. - **Retention: 90d** for reptide events (privacy doc allows ≤90d), fleet journals stay at 48h. - **Public vhost becomes push-only.** Every automated consumer (fleet Alloy, ansible hosts) only pushes; Grafana queries via localhost; nothing uses the public query API. Lock `loki.cloonar.com` to `location = /loki/api/v1/push` (basic auth, existing htpasswd) + `/ready` (open); everything else denied. Closes the read-hole for any leaked pusher credential, reptide's or the fleet's. - **No new secret.** The `reptide-prod` htpasswd line is added to the existing `promtail-nginx-password` secret in `hosts/web-arm/secrets.sops.yaml` (operator edit via sops; same password goes to Supabase as `LOKI_PASSWORD` per MANUAL_SETUP.md §2–3). Deploy order independent of the secret edit. - **Grafana: alert only, no dashboard.** Explore covers early triage; a dashboard can be frozen into Nix after the first real incident shows what's useful. ### Work (single PR, two commits) 1. `hosts/web-arm/modules/loki.nix` - `limits_config.retention_stream = [{ selector = ''{app="reptide"}''; priority = 1; period = "2160h"; }]` (compactor already has `retention_enabled = true`) - nginx vhost: exact-match push location with existing auth, `/ready` open, deny the rest 2. `hosts/web-arm/modules/grafana/alerting/reptide/` — error-rate alert per MANUAL_SETUP.md §8: `sum(rate({app="reptide", env="production", level="error"}[5m])) > 3 * <24h baseline offset 24h>`, sustained 10m, routed via existing default policy → `cp_dominik_normal` (Pushover; the Matrix room in the doc is reptide-side wiring, not available here). ### Verification - agent, post-deploy: unauthenticated surface (push → 401, query paths → 403/404, `/ready` → 200); fleet journals still flowing; `Coredumps` ruler alert intact; retention override in rendered config; look into `/ready` currently returning 503 (unmonitored — blackbox blacklists the domain) - operator: add `reptide-prod` htpasswd line, then authenticated push → 204; `supabase secrets set` + edge-function deploy per MANUAL_SETUP.md; `{app="reptide"}` visible in Grafana Explore

dominik.polakovics added the

ready-for-agent

label

2026-08-10 23:02:37 +02:00

dominik.polakovics referenced this issue

2026-08-10 23:13:22 +02:00

matrix-hookshot on the web VM: inbound webhook bridge for reptide feedback channel #324

dominik.polakovics referenced this issue from a commit

2026-08-10 23:32:56 +02:00

feat(web-arm): keep reptide streams 90d and lock the Loki vhost to push-only

dominik.polakovics referenced this issue from a commit

2026-08-10 23:32:56 +02:00