feat(metrics): evaluate consolidating metrics shipping into grafana-alloy (unify with logs) #120

New issue

Open

opened 2026-06-07 14:20:18 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-06-07 14:20:18 +02:00

Owner

Deferred follow-up to #118 (promtail→alloy logs migration). Not urgent.

Agent Brief

Category: enhancement

Summary: Evaluate folding the per-host metrics shipping (vmagent-style scraping → VictoriaMetrics) into grafana-alloy, so a single Alloy agent per host handles both journal→Loki logs and metrics→VictoriaMetrics fleet-wide.

Background / why: #118 migrates journal shipping from promtail to grafana-alloy, deliberately scoped to logs only. Alloy is a superset — it can prometheus.scrape and prometheus.remote_write — so once every host already runs Alloy for logs, the metrics path could be expressed in the same agent, leaving one observability agent per host instead of two (Alloy + vmagent). This was explicitly deferred out of #118 to avoid scope-creep on a forced, time-sensitive migration: metrics aren't broken, so there's no forcing function.

Current behavior: Metrics are a separate subsystem — utils/modules/victoriametrics (shared, with a monitoredServices option) on fw/nas/amzebs-01, hosts/web-arm/modules/victoriametrics.nix on web-arm, plus per-host exporters (mail dovecot/postfix, fw fwmetrics, nas disk-monitoring). After #118 every host also runs grafana-alloy for logs.

Desired behavior: A spike + decision on whether to re-express the existing scrape targets (monitoredServices + each exporter) and remote_write to VictoriaMetrics as Alloy components, validated against the existing Grafana dashboards and alerting on web-arm. If worthwhile, migrate host-by-host and retire vmagent. Acceptance: dashboards/alerts unchanged, single agent per host.

Dependencies: Blocked on #118 (Alloy must be fleet-wide first).

*Deferred follow-up to #118 (promtail→alloy logs migration). Not urgent.* ## Agent Brief **Category:** enhancement **Summary:** Evaluate folding the per-host metrics shipping (vmagent-style scraping → VictoriaMetrics) into grafana-alloy, so a single Alloy agent per host handles both journal→Loki logs *and* metrics→VictoriaMetrics fleet-wide. **Background / why:** #118 migrates journal shipping from promtail to grafana-alloy, deliberately scoped to **logs only**. Alloy is a superset — it can `prometheus.scrape` and `prometheus.remote_write` — so once every host already runs Alloy for logs, the metrics path could be expressed in the same agent, leaving one observability agent per host instead of two (Alloy + vmagent). This was explicitly deferred out of #118 to avoid scope-creep on a forced, time-sensitive migration: metrics aren't broken, so there's no forcing function. **Current behavior:** Metrics are a separate subsystem — `utils/modules/victoriametrics` (shared, with a `monitoredServices` option) on fw/nas/amzebs-01, `hosts/web-arm/modules/victoriametrics.nix` on web-arm, plus per-host exporters (mail dovecot/postfix, fw fwmetrics, nas disk-monitoring). After #118 every host also runs grafana-alloy for logs. **Desired behavior:** A spike + decision on whether to re-express the existing scrape targets (`monitoredServices` + each exporter) and remote_write to VictoriaMetrics as Alloy components, validated against the existing Grafana dashboards and alerting on web-arm. If worthwhile, migrate host-by-host and retire vmagent. Acceptance: dashboards/alerts unchanged, single agent per host. **Dependencies:** Blocked on #118 (Alloy must be fleet-wide first).

dominik.polakovics added the

needs-triage

enhancement

labels

2026-06-07 14:20:18 +02:00

dominik.polakovics referenced this issue

2026-06-07 14:33:14 +02:00

feat(logging): grafana-alloy shared module + nas canary (promtail→alloy step 1) #121

dominik.polakovics referenced this issue

2026-06-07 14:50:36 +02:00

feat(logging): migrate fleet journal shipping from promtail to grafana-alloy #118