Cloonar/nixos

Fork 0

feat(web-arm): channel → nixos-26.05 [upgrade 5/6 · bump] #109

New issue

Closed

opened 2026-06-06 12:07:03 +02:00 by dominik.polakovics · 2 comments

dominik.polakovics commented

2026-06-06 12:07:03 +02:00

Owner

▶ Hold until [4/6 · verify] is closed. Then arm: relabel ready-for-agent.

Task: bump hosts/web-arm/channel to nixos-26.05; open PR.

26.05 specifics for web-arm:

Grafana secret_key no longer has a default. web-arm does not currently set it → add services.grafana.settings.security.secret_key, wired to SOPS, value = the old default SW2YcwTIb9zpOOhoPsMm to preserve already-encrypted datasource credentials (a new key would break them). Human must populate the SOPS secret — do not edit secrets files directly; list the key + where in the PR.
PostgreSQL is pinned to postgresql_14 → no forced major migration; confirm the attr still exists on 26.05 (eval).
PowerSync is self-hosted (ADR-0012/0015, custom module) — confirm it still builds on 26.05.
aarch64 host: eval is arch-agnostic; the full build runs on the host at deploy.
stateVersion stays 22.05.

Acceptance: pre-commit eval green; grafana secret_key wired to SOPS (value to be populated by human).

▶ **Hold** until [4/6 · verify] is closed. Then arm: relabel `ready-for-agent`. **Task:** bump `hosts/web-arm/channel` to `nixos-26.05`; open PR. **26.05 specifics for web-arm:** - **Grafana `secret_key`** no longer has a default. web-arm does not currently set it → add `services.grafana.settings.security.secret_key`, wired to **SOPS**, value = the **old default `SW2YcwTIb9zpOOhoPsMm`** to preserve already-encrypted datasource credentials (a *new* key would break them). **Human must populate the SOPS secret** — do not edit secrets files directly; list the key + where in the PR. - **PostgreSQL** is pinned to `postgresql_14` → no forced major migration; confirm the attr still exists on 26.05 (eval). - **PowerSync** is self-hosted (ADR-0012/0015, custom module) — confirm it still builds on 26.05. - aarch64 host: eval is arch-agnostic; the full build runs on the host at deploy. - stateVersion stays `22.05`. **Acceptance:** pre-commit eval green; grafana secret_key wired to SOPS (value to be populated by human).

dominik.polakovics added the

needs-triage

label

2026-06-06 12:07:03 +02:00

dominik.polakovics referenced this issue

2026-06-07 12:53:04 +02:00

feat(logging): migrate fleet journal shipping from promtail to grafana-alloy #118

dominik.polakovics commented

2026-06-07 15:09:54 +02:00

Author

Owner

This was generated by AI during triage.

Scope addition: drop promtail on the 26.05 bump (web-arm specifics)

26.05 removed services.promtail, so this bump must drop web-arm's ./utils/modules/promtail CLIENT import (same as nb/nas), with a paused-shipping comment pointing at the separate promtail->alloy migration (#118). Alloy setup is not in scope.

web-arm is special: it also runs the Loki server (modules/loki.nix) -- the receiving end -- with promtail-nginx-password basic-auth for clients pushing to loki.cloonar.com. That is the server side, not the promtail client: leave Loki and its promtail-nginx-password in place for now (the alloy migration #118 redirects clients to it). Only the client module import comes out here.

Also drop web-arm's temporary docker_29 pin and the sa-core docker redirect (from #123) -- redundant on 26.05.

Full agent brief will be posted when this issue is armed.

> *This was generated by AI during triage.* ## Scope addition: drop promtail on the 26.05 bump (web-arm specifics) 26.05 **removed `services.promtail`**, so this bump must drop web-arm's `./utils/modules/promtail` CLIENT import (same as nb/nas), with a paused-shipping comment pointing at the separate promtail->alloy migration (#118). Alloy setup is not in scope. **web-arm is special:** it also runs the **Loki server** (`modules/loki.nix`) -- the receiving end -- with `promtail-nginx-password` basic-auth for clients pushing to `loki.cloonar.com`. That is the server side, not the promtail client: leave Loki and its `promtail-nginx-password` in place for now (the alloy migration #118 redirects clients to it). Only the client module import comes out here. Also drop web-arm's temporary `docker_29` pin and the `sa-core` docker redirect (from #123) -- redundant on 26.05. Full agent brief will be posted when this issue is armed.

dominik.polakovics referenced this issue

2026-06-08 12:29:11 +02:00

mail: reboot onto 26.05 + verify [upgrade 4/6 · verify] #108

dominik.polakovics commented

2026-06-08 12:56:21 +02:00

Author

Owner

This was generated by AI during triage.

Agent Brief

Category: enhancement
Summary: Bump web-arm's channel to nixos-26.05 (5/6 of the staged fleet upgrade) and fix what 26.05 surfaces; pre-commit eval must stay green. stateVersion stays 22.05.

Current behavior:
web-arm tracks https://channels.nixos.org/nixos-25.11, carries two 25.11-era workarounds 26.05 makes obsolete (docker_29 pin, permittedInsecurePackages), uses the pre-rename ACME credentialsFile option, and never sets grafana's secret_key (inheriting the 25.11 module default).

Desired behavior:
web-arm tracks nixos-26.05, evaluates green under the pre-commit dry-build, and preserves all current runtime behavior (TLS issuance, grafana datasource-secret decryption, docker scanners, PowerSync replication).

26.05 changes to handle:

permittedInsecurePackages → allowInsecurePredicate. 26.05's makePythonWriter interpreter guard force-evaluates the whole pypy2Packages set (via fetch-cargo-vendor-util, pulled in by the users-groups shell-program assertion), tripping the now-insecure pypy2.7-* members. Defining an allowInsecurePredicate disables the permittedInsecurePackages list, so the existing openssl-1.1.1v / openssl-1.1.1w allowances must fold into the same predicate. Mirror nb/nas/fw: hasPrefix "pypy2.7-" OR elem name ["openssl-1.1.1v" "openssl-1.1.1w"], with the pkg.name or "${pkg.pname}-${pkg.version}" fallback. (If 26.05 flags any other web-arm package insecure/unfree, fold it in minimally; none expected.)
Drop the docker_29 pin. PR #123 pinned virtualisation.docker.package = pkgs.docker_29 to dodge the EOL/insecure docker_28 default on 25.11; 26.05's default docker is maintained, so remove the pin + comment. The sa-core (ScanA11y) scanner units redirect their path through config.virtualisation.docker.package only to follow that pin — revert to the default pkgs.docker and drop the stale "pinned to docker_29 / 25.11 docker_28" comment.
ACME credentialsFile → environmentFile. 26.05 dropped the credentialsFile rename alias on the acme cert submodule. The shared lego.nix default was already migrated; web-arm additionally sets security.acme.certs.<name>.credentialsFile on several per-site certs and in the powersync module. Migrate every such usage to environmentFile = … (same sops-secret value). environmentFile is valid on both channels, so it's safe. (Grep credentialsFile under hosts/web-arm — ~12 hits across the per-site cert defs and the powersync module.)
Grafana secret_key (26.05 asserts it — eval-blocking). 26.05 removed the default for services.grafana.settings.security.secret_key and adds a hard assertion secret_key != null, so the dry-build fails until it's set. web-arm currently inherits the 25.11 module default SW2YcwTIb9zpOOhoPsMm, which is the key its grafana DB is already encrypted with — preserve that exact value so no encrypted-at-rest state is orphaned. Wire it via the file-provider (not inline — inline lands world-readable in the store, and the assertion message says use a file-provider), mirroring the existing admin_password usage: set security.secret_key = "$__file{${config.sops.secrets.grafana-secret-key.path}}" and declare sops.secrets.grafana-secret-key.owner = "grafana". The grafana-secret-key secret already exists in web-arm's secrets.yaml (value = the old default), so no secret editing is needed — just declare + reference it. The $__file{} literal is non-null, so the assertion passes at eval; grafana resolves the file at runtime.
postgresql_14 pin — confirm, do NOT migrate. web-arm pins services.postgresql.package = pkgs.postgresql_14 deliberately (PowerSync bucket store + every app DB live in it; a major bump is a data migration). Confirm the attr still resolves on 26.05 (PG14 EOL is Nov 2026, so expected present). If it's gone, STOP and escalate — do not auto-bump the major.

Acceptance criteria:

hosts/web-arm/channel = https://channels.nixos.org/nixos-26.05
permittedInsecurePackages replaced by allowInsecurePredicate covering pypy2.7-* + openssl-1.1.1v + openssl-1.1.1w
docker_29 pin removed; sa-core path reverted to default pkgs.docker; stale docker comments gone
every per-cert credentialsFile under web-arm migrated to environmentFile
grafana security.secret_key wired to the existing grafana-secret-key sops secret (owner grafana) via $__file{…}
postgresql_14 confirmed resolvable on 26.05 (else escalated, not migrated)
system.stateVersion unchanged (22.05)
pre-commit dry-build (eval) green for web-arm
PR opened, Closes #109

Out of scope:

promtail: already removed (alloy fan-out #125 / b963e65). No client import to drop. Leave the Loki SERVER (loki.nix) and promtail-nginx-password (server-side basic auth) untouched. This supersedes the 2026-06-07 comment's promtail instruction.
The shared lego.nix default (already migrated).
Any PostgreSQL major migration; any grafana secret_key rotation to a strong key (a separate, deliberate step).
Build/runtime verification — web-arm is aarch64, the full build runs on the host at deploy; eval can't catch build/runtime 26.05 breakage in web-arm's unique services (nextcloud, collabora, mysql, matomo, typo3, bitwarden, rustdesk, atticd, victoriametrics, powersync). That's the [5/6 · verify] job (#110), not this bump.

> *This was generated by AI during triage.* ## Agent Brief **Category:** enhancement **Summary:** Bump web-arm's channel to nixos-26.05 (5/6 of the staged fleet upgrade) and fix what 26.05 surfaces; pre-commit eval must stay green. stateVersion stays 22.05. **Current behavior:** web-arm tracks `https://channels.nixos.org/nixos-25.11`, carries two 25.11-era workarounds 26.05 makes obsolete (docker_29 pin, permittedInsecurePackages), uses the pre-rename ACME `credentialsFile` option, and never sets grafana's `secret_key` (inheriting the 25.11 module default). **Desired behavior:** web-arm tracks nixos-26.05, evaluates green under the pre-commit dry-build, and preserves all current runtime behavior (TLS issuance, grafana datasource-secret decryption, docker scanners, PowerSync replication). **26.05 changes to handle:** 1. **permittedInsecurePackages → allowInsecurePredicate.** 26.05's makePythonWriter interpreter guard force-evaluates the whole pypy2Packages set (via fetch-cargo-vendor-util, pulled in by the users-groups shell-program assertion), tripping the now-insecure pypy2.7-* members. Defining an allowInsecurePredicate disables the permittedInsecurePackages list, so the existing openssl-1.1.1v / openssl-1.1.1w allowances must fold into the same predicate. Mirror nb/nas/fw: `hasPrefix "pypy2.7-"` OR `elem name ["openssl-1.1.1v" "openssl-1.1.1w"]`, with the `pkg.name or "${pkg.pname}-${pkg.version}"` fallback. (If 26.05 flags any other web-arm package insecure/unfree, fold it in minimally; none expected.) 2. **Drop the docker_29 pin.** PR #123 pinned `virtualisation.docker.package = pkgs.docker_29` to dodge the EOL/insecure docker_28 default on 25.11; 26.05's default docker is maintained, so remove the pin + comment. The sa-core (ScanA11y) scanner units redirect their `path` through `config.virtualisation.docker.package` only to follow that pin — revert to the default `pkgs.docker` and drop the stale "pinned to docker_29 / 25.11 docker_28" comment. 3. **ACME credentialsFile → environmentFile.** 26.05 dropped the `credentialsFile` rename alias on the acme cert submodule. The shared lego.nix default was already migrated; web-arm additionally sets `security.acme.certs.<name>.credentialsFile` on several per-site certs and in the powersync module. Migrate every such usage to `environmentFile = …` (same sops-secret value). `environmentFile` is valid on both channels, so it's safe. (Grep `credentialsFile` under hosts/web-arm — ~12 hits across the per-site cert defs and the powersync module.) 4. **Grafana secret_key (26.05 asserts it — eval-blocking).** 26.05 removed the default for `services.grafana.settings.security.secret_key` and adds a hard assertion `secret_key != null`, so the dry-build fails until it's set. web-arm currently inherits the 25.11 module default `SW2YcwTIb9zpOOhoPsMm`, which is the key its grafana DB is already encrypted with — preserve that exact value so no encrypted-at-rest state is orphaned. Wire it via the file-provider (not inline — inline lands world-readable in the store, and the assertion message says use a file-provider), mirroring the existing `admin_password` usage: set `security.secret_key = "$__file{${config.sops.secrets.grafana-secret-key.path}}"` and declare `sops.secrets.grafana-secret-key.owner = "grafana"`. The `grafana-secret-key` secret already exists in web-arm's secrets.yaml (value = the old default), so no secret editing is needed — just declare + reference it. The `$__file{}` literal is non-null, so the assertion passes at eval; grafana resolves the file at runtime. 5. **postgresql_14 pin — confirm, do NOT migrate.** web-arm pins `services.postgresql.package = pkgs.postgresql_14` deliberately (PowerSync bucket store + every app DB live in it; a major bump is a data migration). Confirm the attr still resolves on 26.05 (PG14 EOL is Nov 2026, so expected present). If it's gone, STOP and escalate — do not auto-bump the major. **Acceptance criteria:** - [ ] hosts/web-arm/channel = https://channels.nixos.org/nixos-26.05 - [ ] permittedInsecurePackages replaced by allowInsecurePredicate covering pypy2.7-* + openssl-1.1.1v + openssl-1.1.1w - [ ] docker_29 pin removed; sa-core path reverted to default pkgs.docker; stale docker comments gone - [ ] every per-cert credentialsFile under web-arm migrated to environmentFile - [ ] grafana security.secret_key wired to the existing grafana-secret-key sops secret (owner grafana) via $__file{…} - [ ] postgresql_14 confirmed resolvable on 26.05 (else escalated, not migrated) - [ ] system.stateVersion unchanged (22.05) - [ ] pre-commit dry-build (eval) green for web-arm - [ ] PR opened, Closes #109 **Out of scope:** - promtail: already removed (alloy fan-out #125 / b963e65). No client import to drop. Leave the Loki SERVER (loki.nix) and promtail-nginx-password (server-side basic auth) untouched. This supersedes the 2026-06-07 comment's promtail instruction. - The shared lego.nix default (already migrated). - Any PostgreSQL major migration; any grafana secret_key rotation to a strong key (a separate, deliberate step). - Build/runtime verification — web-arm is aarch64, the full build runs on the host at deploy; eval can't catch build/runtime 26.05 breakage in web-arm's unique services (nextcloud, collabora, mysql, matomo, typo3, bitwarden, rustdesk, atticd, victoriametrics, powersync). That's the [5/6 · verify] job (#110), not this bump.

dominik.polakovics added

ready-for-agent

and removed

needs-triage

labels

2026-06-08 12:57:13 +02:00

dominik.polakovics referenced this issue from a commit

2026-06-08 13:07:16 +02:00

feat(web-arm): channel → nixos-26.05

dominik.polakovics referenced this issue from a pull request that will close it,

2026-06-08 13:08:12 +02:00

feat(web-arm): channel → nixos-26.05 #141