Add Grafana alert on the sync_quarantine Loki stream (Reptide silent sync drops) #172

New issue

Open

opened 2026-06-22 23:10:03 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-06-22 23:10:03 +02:00

Owner

Context

The Reptide app (Cloonar/fit) now emits a telemetry event to Loki whenever the PowerSync connector drops a permanently-rejected upload op — a row the server rejects on every retry (FK / unique / RLS), quarantined so one bad write can't wedge the whole offline-sync queue. See fit PR Cloonar/fit#379 and ADR docs/adr/0048-connector-referential-upload-ordering.md.

A quarantine drop unblocks the user's sync but silently loses that one write. In steady state it should be ~zero; a nonzero / rising rate means either a genuine data condition or a new bug systematically dropping rows. The quarantine is a circuit breaker — this issue is its gauge: a Grafana alert on the drop rate so a spike is team-visible instead of silent.

Telemetry contract (already shipping from fit)

Client events reach Loki via the log-event edge function. A quarantine event is a Loki stream with these labels:

label	value
`app`	`reptide`
`category`	`sync_quarantine`
`level`	`warn`
`env`	`production` (from `LOKI_ENV`)
`platform`	`ios` / `android` / …
`version`, `build`, `git_commit`	build coordinates

The JSON log line carries message (human summary), runtimeType = "sync_quarantine", and extra = { table, op, sqlstate, row_id } — the dropped row's coordinates (e.g. table="template_sets", op="put", sqlstate="42501").

Ask

Add a provisioned Grafana alert rule (on web-arm) that fires when the sync_quarantine drop rate over a window exceeds a threshold.

Stream selector: {app="reptide", category="sync_quarantine"}

Suggested query (Loki datasource loki-datasource-uid):

sum(count_over_time({app="reptide", category="sync_quarantine"}[15m]))

Suggested starting threshold (tunable): warning if the 15m count >= 3 with for = "10m" — i.e. sustained drops, not a one-off. Consider splitting by env/platform (sum by (env) (...)). A single isolated drop happens occasionally; a sustained or spiking rate is the real signal.

Implementation pointers (mirror the existing pattern)

Alerts live under hosts/web-arm/modules/grafana/alerting/<category>/: each rule is a grafanaAlertRuleDefinitions list, aggregated by a default.nix into a services.grafana.provision.alerting.rules.settings.groups group, and imported in hosts/web-arm/modules/grafana/default.nix (imports, ~lines 30–34).

Add e.g. alerting/app/sync_quarantine.nix (a new app/reptide category folder, or fold into service/), plus its default.nix, and wire it into grafana/default.nix imports.
First-of-its-kind note: every existing alert queries Prometheus/VictoriaMetrics (vm-datasource-uid). This would be the first Loki-datasource alert. Use datasourceUid = "loki-datasource-uid" (see grafana/datasources/loki.nix) with queryType = "range", then the same reduce(last) → math($X > threshold) expression chain as alerting/system/ram_usage.nix.

Routing

No contact-point change required: the root notification policy routes all alerts to cp_dominik_normal (Pushover, priority 1). If sync data-loss should page louder, add a route matching this rule's alertname to cp_dominik_emergency, mirroring the existing HostDown route in grafana/default.nix.

Acceptance criteria

A provisioned Grafana alert rule querying the Loki sync_quarantine stream, deployed on web-arm.
Fires on a sustained / spiking drop rate; resolves when it returns to ~0.
Routes to a contact point (default cp_dominik_normal is fine).
Threshold + window live in the rule and are easy to tune.
(Nice-to-have) A Grafana panel breaking sync_quarantine down by table / sqlstate (parsed from the log line's extra) for triage.

## Context The Reptide app (`Cloonar/fit`) now emits a telemetry event to Loki whenever the PowerSync connector **drops a permanently-rejected upload op** — a row the server rejects on every retry (FK / unique / RLS), quarantined so one bad write can't wedge the whole offline-sync queue. See fit PR https://git.cloonar.com/Cloonar/fit/pulls/379 and ADR `docs/adr/0048-connector-referential-upload-ordering.md`. A quarantine drop unblocks the user's sync **but silently loses that one write**. In steady state it should be ~zero; a nonzero / rising rate means either a genuine data condition or a new bug systematically dropping rows. The quarantine is a circuit breaker — this issue is its **gauge**: a Grafana alert on the drop rate so a spike is team-visible instead of silent. ## Telemetry contract (already shipping from fit) Client events reach Loki via the `log-event` edge function. A quarantine event is a Loki stream with these labels: | label | value | |---|---| | `app` | `reptide` | | `category` | `sync_quarantine` | | `level` | `warn` | | `env` | `production` (from `LOKI_ENV`) | | `platform` | `ios` / `android` / … | | `version`, `build`, `git_commit` | build coordinates | The JSON log line carries `message` (human summary), `runtimeType = "sync_quarantine"`, and `extra = { table, op, sqlstate, row_id }` — the dropped row's coordinates (e.g. `table="template_sets", op="put", sqlstate="42501"`). ## Ask Add a provisioned Grafana alert rule (on `web-arm`) that fires when the `sync_quarantine` drop rate over a window exceeds a threshold. **Stream selector:** `{app="reptide", category="sync_quarantine"}` **Suggested query (Loki datasource `loki-datasource-uid`):** ```logql sum(count_over_time({app="reptide", category="sync_quarantine"}[15m])) ``` **Suggested starting threshold (tunable):** warning if the 15m count `>= 3` with `for = "10m"` — i.e. sustained drops, not a one-off. Consider splitting by env/platform (`sum by (env) (...)`). A single isolated drop happens occasionally; a sustained or spiking rate is the real signal. ## Implementation pointers (mirror the existing pattern) Alerts live under `hosts/web-arm/modules/grafana/alerting/<category>/`: each rule is a `grafanaAlertRuleDefinitions` list, aggregated by a `default.nix` into a `services.grafana.provision.alerting.rules.settings.groups` group, and imported in `hosts/web-arm/modules/grafana/default.nix` (`imports`, ~lines 30–34). - Add e.g. `alerting/app/sync_quarantine.nix` (a new `app`/`reptide` category folder, or fold into `service/`), plus its `default.nix`, and wire it into `grafana/default.nix` imports. - **First-of-its-kind note:** every existing alert queries Prometheus/VictoriaMetrics (`vm-datasource-uid`). This would be the **first Loki-datasource alert**. Use `datasourceUid = "loki-datasource-uid"` (see `grafana/datasources/loki.nix`) with `queryType = "range"`, then the same `reduce`(last) → `math`(`$X > threshold`) expression chain as `alerting/system/ram_usage.nix`. ## Routing No contact-point change required: the root notification policy routes all alerts to `cp_dominik_normal` (Pushover, priority 1). If sync data-loss should page louder, add a route matching this rule's `alertname` to `cp_dominik_emergency`, mirroring the existing `HostDown` route in `grafana/default.nix`. ## Acceptance criteria - [ ] A provisioned Grafana alert rule querying the Loki `sync_quarantine` stream, deployed on `web-arm`. - [ ] Fires on a sustained / spiking drop rate; resolves when it returns to ~0. - [ ] Routes to a contact point (default `cp_dominik_normal` is fine). - [ ] Threshold + window live in the rule and are easy to tune. - [ ] (Nice-to-have) A Grafana panel breaking `sync_quarantine` down by `table` / `sqlstate` (parsed from the log line's `extra`) for triage.

dominik.polakovics added the

needs-triage

label

2026-06-22 23:10:03 +02:00