feat(fw): channel → nixos-26.05 [upgrade 3/6 · bump] #105

Closed
opened 2026-06-06 12:07:02 +02:00 by dominik.polakovics · 4 comments

Hold until [2/6 · verify] is closed. Then arm: relabel ready-for-agent.

Task: bump hosts/fw/channel to nixos-26.05; open PR.

26.05 specifics for fw (keystone):

  • This bumps fw and all 5 guest microVMs (dev/lab, forgejo-runner, openclaw, web-02) — their nixpkgs track fw's. microvm has no 26.05 breaking change flagged.
  • nixpkgs.config.permittedInsecurePackages — confirm the pinned package names still resolve on 26.05 (eval will flag a stale name).
  • Kernel 6.12→6.18: router NIC drivers are the reboot risk (verified at [3/6 · verify], not here).
  • stateVersion stays 22.05.

Acceptance: pre-commit eval green for fw and its guests against 26.05.

▶ **Hold** until [2/6 · verify] is closed. Then arm: relabel `ready-for-agent`. **Task:** bump `hosts/fw/channel` to `nixos-26.05`; open PR. **26.05 specifics for fw (keystone):** - This bumps **fw and all 5 guest microVMs** (dev/lab, forgejo-runner, openclaw, web-02) — their nixpkgs track fw's. `microvm` has **no** 26.05 breaking change flagged. - `nixpkgs.config.permittedInsecurePackages` — confirm the pinned package names still resolve on 26.05 (eval will flag a stale name). - Kernel 6.12→6.18: **router NIC drivers** are the reboot risk (verified at [3/6 · verify], not here). - stateVersion stays `22.05`. **Acceptance:** pre-commit eval green for fw **and** its guests against 26.05.
Author
Owner

This was generated by AI during triage.

Agent Brief

Category: enhancement (channel upgrade)
Summary: Bump fw and its 5 guest microVMs to nixos-26.05 -- flip the host channel, drop the now-removed promtail module and the now-redundant docker_29 pin, and fix whatever else 26.05 surfaces. Leave eval green for fw AND every guest, and a real build clean.

Current behavior:
fw tracks nixos-25.11 and is the keystone: 5 microVM guests (dev, web/web-02, forgejo-runner, openclaw) track its nixpkgs, plus routing/DNS (dnsmasq), Forgejo (the deploy substrate), home-assistant, foundry, podman/oci containers. system.stateVersion = 22.05.

  • Imports the shared promtail module (journald->loki shipping).
  • Its dev guest pins virtualisation.docker.package = pkgs.docker_29 as a temporary 25.11 workaround (docker comes from utils/modules/development; the 25.11 default docker_28 is EOL/insecure).
  • permittedInsecurePackages = ["openssl-1.1.1w"] on the host; the web guest pins its own ["olm-3.2.16"] (matrix bridges).
  • boot.kernelPackages pins linuxPackages_6_18 behind versionOlder ... "6.18.22".

Desired behavior:
fw and all 5 guests evaluate and BUILD cleanly on nixos-26.05, services unchanged. Diff = channel flip + the forced 26.05 fixes below.

Known 26.05 specifics for fw:

  • Drop promtail (mandatory): 26.05 removed services.promtail (EOL), so the ./utils/modules/promtail import must go or eval fails on an unknown option. Mirror nas: replace the import with a short comment noting central shipping is paused pending the SEPARATE promtail->grafana-alloy migration (#118; sops scaffolding already exists under utils/modules/alloy/, inert until #121). Do NOT set up alloy here -- that is #118's scope.
  • Drop the docker_29 pin: the virtualisation.docker.package = pkgs.docker_29 workaround on the dev guest is redundant on 26.05 (default docker is maintained). Remove it.
  • Insecure pins: re-validate openssl-1.1.1w (host) and olm-3.2.16 (web guest) still resolve and are still needed on 26.05; eval flags stale names. Prefer fixing over carrying a dead pin.
  • Highest-churn guests: the web guest runs matrix-synapse + matrix-authentication-service (+ mautrix/olm), invidious (+companion), and piped -- the likeliest sources of 26.05 option renames.
  • Kernel pin: confirm linuxPackages_6_18 self-disables (default >= 6.18.22) or still resolves. NIC-driver reboot risk is verified at #106, not here.

Acceptance criteria:

  • hosts/fw/channel -> nixos-26.05.
  • Pre-commit eval green for fw AND every guest microVM (dev, web, forgejo-runner, openclaw) on 26.05.
  • A real fw system-closure BUILD succeeds -- eval-only pre-commit misses 26.05 build-gates (udevadm verify, broken pkgs); the nb udev-% regression only surfaced post-merge.
  • promtail import removed; docker_29 pin removed; no host imports utils/modules/alloy yet.
  • system.stateVersion stays 22.05; diff limited to channel flip + required fixes.
  • Any new insecure/unfree allowance justified in the PR (proper fix preferred over blanket masking).

Out of scope:

  • The promtail->alloy migration itself (#118 / #121) -- only the import removal belongs here.
  • The keystone reboot + runtime verification (DNS, Forgejo, all 5 guests) -- that is #106 (ready-for-human).
  • Removing the redundant linuxPackages_6_18 pin (separate cleanup).
  • Other hosts' channels.
> *This was generated by AI during triage.* ## Agent Brief **Category:** enhancement (channel upgrade) **Summary:** Bump fw and its 5 guest microVMs to `nixos-26.05` -- flip the host `channel`, drop the now-removed promtail module and the now-redundant docker_29 pin, and fix whatever else 26.05 surfaces. Leave eval green for fw AND every guest, and a real build clean. **Current behavior:** fw tracks `nixos-25.11` and is the keystone: 5 microVM guests (dev, web/web-02, forgejo-runner, openclaw) track its nixpkgs, plus routing/DNS (dnsmasq), Forgejo (the deploy substrate), home-assistant, foundry, podman/oci containers. `system.stateVersion` = 22.05. - Imports the shared `promtail` module (journald->loki shipping). - Its dev guest pins `virtualisation.docker.package = pkgs.docker_29` as a temporary 25.11 workaround (docker comes from `utils/modules/development`; the 25.11 default docker_28 is EOL/insecure). - `permittedInsecurePackages = ["openssl-1.1.1w"]` on the host; the web guest pins its own `["olm-3.2.16"]` (matrix bridges). - `boot.kernelPackages` pins `linuxPackages_6_18` behind `versionOlder ... "6.18.22"`. **Desired behavior:** fw and all 5 guests evaluate and BUILD cleanly on `nixos-26.05`, services unchanged. Diff = channel flip + the forced 26.05 fixes below. **Known 26.05 specifics for fw:** - **Drop promtail (mandatory):** 26.05 removed `services.promtail` (EOL), so the `./utils/modules/promtail` import must go or eval fails on an unknown option. Mirror nas: replace the import with a short comment noting central shipping is paused pending the SEPARATE promtail->grafana-alloy migration (#118; sops scaffolding already exists under `utils/modules/alloy/`, inert until #121). Do NOT set up alloy here -- that is #118's scope. - **Drop the docker_29 pin:** the `virtualisation.docker.package = pkgs.docker_29` workaround on the dev guest is redundant on 26.05 (default docker is maintained). Remove it. - **Insecure pins:** re-validate `openssl-1.1.1w` (host) and `olm-3.2.16` (web guest) still resolve and are still needed on 26.05; eval flags stale names. Prefer fixing over carrying a dead pin. - **Highest-churn guests:** the web guest runs `matrix-synapse` + `matrix-authentication-service` (+ mautrix/olm), `invidious` (+companion), and `piped` -- the likeliest sources of 26.05 option renames. - **Kernel pin:** confirm `linuxPackages_6_18` self-disables (default >= 6.18.22) or still resolves. NIC-driver reboot risk is verified at #106, not here. **Acceptance criteria:** - [ ] `hosts/fw/channel` -> `nixos-26.05`. - [ ] Pre-commit eval green for fw AND every guest microVM (dev, web, forgejo-runner, openclaw) on 26.05. - [ ] A real fw system-closure BUILD succeeds -- eval-only pre-commit misses 26.05 build-gates (`udevadm verify`, broken pkgs); the nb udev-`%` regression only surfaced post-merge. - [ ] promtail import removed; docker_29 pin removed; no host imports `utils/modules/alloy` yet. - [ ] `system.stateVersion` stays 22.05; diff limited to channel flip + required fixes. - [ ] Any new insecure/unfree allowance justified in the PR (proper fix preferred over blanket masking). **Out of scope:** - The promtail->alloy migration itself (#118 / #121) -- only the import removal belongs here. - The keystone reboot + runtime verification (DNS, Forgejo, all 5 guests) -- that is #106 (ready-for-human). - Removing the redundant `linuxPackages_6_18` pin (separate cleanup). - Other hosts' channels.
Author
Owner

This was generated by AI during triage.

Agent Brief (REVISED — supersedes the brief above)

Revised after the promtail→alloy migration (#118/#124/#125) landed and after re-checking fw against 26.05. Net change vs the earlier brief: the promtail/alloy work is already done (removed), and the one genuinely-forced 26.05 fix is the insecure-packages predicate migration (added).

Category: enhancement (channel upgrade)
Summary: Bump fw and its 5 guest microVMs to nixos-26.05 — flip the host channel, drop the now-redundant docker_29 pin, migrate the host and guests from permittedInsecurePackages to allowInsecurePredicate (26.05 forces this), and fix anything else 26.05 surfaces. Minimal diff: channel flip + forced fixes only.

Current behavior:
fw tracks nixos-25.11 and is the keystone. 5 microVM guests track its nixpkgs — dev, web (web-02), fj-runner-1, fj-runner-2 (forgejo-runner declares two via mapAttrs), openclaw — plus routing/DNS (dnsmasq), Forgejo (the deploy substrate), home-assistant, foundry, podman/oci. system.stateVersion = 22.05.

  • Host sets nixpkgs.config.permittedInsecurePackages = [ "openssl-1.1.1w" ].
  • The web guest is its own nixpkgs instance (import pkgs.path { config.permittedInsecurePackages = [ "olm-3.2.16" ]; … }) for the matrix bridges (mautrix/olm).
  • The dev guest pins virtualisation.docker.package = pkgs.docker_29 (temporary 25.11 workaround; the 25.11 default docker_28 is EOL/insecure). docker is enabled via shared utils/modules/development.
  • Host + guests apply the shared overlay utils/overlays/packages.nix.
  • boot.kernelPackages pins linuxPackages_6_18 behind lib.versionOlder pkgs.linux.version "6.18.22".
  • Logging already runs on grafana-alloy (promtail→alloy migration complete and live on all hosts). No logging work in this bump.

Desired behavior: fw and all 5 guests evaluate and BUILD cleanly on nixos-26.05, services unchanged. Diff = channel flip + the forced fixes below.

Known 26.05 specifics for fw:

  1. Insecure packages → allowInsecurePredicate (the real forced change). On 26.05 the makePythonWriter interpreter guard force-evaluates the whole pypy2Packages set whenever any Rust package's cargo-vendor derivation is evaluated (pulled in via the users-groups shell-program-check). Its insecure members never enter the closure, but the check trips the insecure gate. Both hosts already on 26.05 (nb, nas) hit this and permit the family by prefix. Because defining allowInsecurePredicate disables the permittedInsecurePackages list, every existing list entry must move into the predicate, per nixpkgs instance:

    • fw host: predicate permitting lib.hasPrefix "pypy2.7-" and openssl-1.1.1w.
    • web guest (separate instance): predicate permitting pypy2.7- and olm-3.2.16 (it builds matrix-authentication-service, which is Rust → trips the same guard).
    • dev / fj-runner-1 / fj-runner-2 / openclaw: add the pypy2.7- predicate to whichever instances eval/build flags (they carry no insecure list today). Use the predicate shape nb/nas use (read pkg.name with a pname-version fallback).
      Prefer the predicate over re-adding a static list — the list silently stops working once a predicate is defined.
  2. Drop the docker_29 pin. On 26.05 the default docker is maintained (proven: nb runs 26.05 with docker via the development module and no pin). Remove the dev guest's virtualisation.docker.package = pkgs.docker_29 and let the default apply.

  3. Do NOT touch pyload. fw's pyload module is disabled (import commented out) and nothing forces pkgs.pyload-ng, so — unlike nas — fw needs no pyload-ng vendoring. The shared utils/overlays/packages.nix still wraps the (26.05-removed) super.pyload-ng, but it's lazy and never forced on fw; leave it alone. Editing the shared overlay would drag every host through the pre-commit dry-build.

  4. Kernel pin is fine. linuxPackages_6_18 is guarded by versionOlder … "6.18.22"; nas runs the identical pin on 26.05. Confirm it resolves/self-disables; don't remove it (separate cleanup).

Acceptance criteria:

  • hosts/fw/channel → nixos-26.05.
  • Pre-commit eval green for fw AND every guest (dev, web, fj-runner-1, fj-runner-2, openclaw) on 26.05.
  • Build-verify: agent BUILDS (not just evals) the high-churn targets on the dev VM — the web guest's matrix stack (matrix-synapse, matrix-authentication-service, mautrix/olm), docker, and anything pulling a Rust cargo-vendor derivation — and reports any closure it could not build locally. Full host+guests build on fw is the maintainer's pre-merge gate (the dev VM can't hold the full keystone closure). eval-only pre-commit misses 26.05 build-gates: udevadm verify, newly-broken pkgs.
  • Host + each guest needing it use allowInsecurePredicate (pypy2.7- + the instance's existing entries); no leftover permittedInsecurePackages that the predicate silently disables.
  • docker_29 pin removed from the dev guest.
  • No logging/promtail/alloy changes; existing utils/modules/alloy imports untouched.
  • pyload untouched; utils/overlays/packages.nix untouched.
  • system.stateVersion stays 22.05; diff limited to channel flip + the fixes above.
  • Any new insecure/unfree allowance justified in the PR.

Out of scope:

  • The promtail→alloy migration (done — #118/#124/#125).
  • The keystone reboot + runtime verification (DNS, Forgejo, all 5 guests, NIC drivers on 6.18) — that's #106 (ready-for-human).
  • Vendoring pyload / editing the shared overlay or shared utils modules.
  • Removing the linuxPackages_6_18 pin (separate cleanup).
  • Other hosts' channels.

What changed vs the brief above: ① removed the entire "drop promtail / set up alloy / no host imports alloy yet" workstream (done); ② added the allowInsecurePredicate + pypy2.7- migration with per-instance fold-in of openssl-1.1.1w/olm-3.2.16; ③ flipped pyload from "mirror nas" to "do NOT vendor"; ④ named all 5 guests (two runners); ⑤ made build-verify concrete.

> *This was generated by AI during triage.* ## Agent Brief (REVISED — supersedes the brief above) *Revised after the promtail→alloy migration (#118/#124/#125) landed and after re-checking fw against 26.05. Net change vs the earlier brief: the promtail/alloy work is **already done** (removed), and the one genuinely-forced 26.05 fix is the **insecure-packages predicate migration** (added).* **Category:** enhancement (channel upgrade) **Summary:** Bump fw and its 5 guest microVMs to nixos-26.05 — flip the host channel, drop the now-redundant docker_29 pin, migrate the host and guests from `permittedInsecurePackages` to `allowInsecurePredicate` (26.05 forces this), and fix anything else 26.05 surfaces. Minimal diff: channel flip + forced fixes only. **Current behavior:** fw tracks nixos-25.11 and is the keystone. 5 microVM guests track its nixpkgs — `dev`, `web` (web-02), `fj-runner-1`, `fj-runner-2` (forgejo-runner declares two via `mapAttrs`), `openclaw` — plus routing/DNS (dnsmasq), Forgejo (the deploy substrate), home-assistant, foundry, podman/oci. `system.stateVersion = 22.05`. - Host sets `nixpkgs.config.permittedInsecurePackages = [ "openssl-1.1.1w" ]`. - The `web` guest is its **own** nixpkgs instance (`import pkgs.path { config.permittedInsecurePackages = [ "olm-3.2.16" ]; … }`) for the matrix bridges (mautrix/olm). - The `dev` guest pins `virtualisation.docker.package = pkgs.docker_29` (temporary 25.11 workaround; the 25.11 default docker_28 is EOL/insecure). docker is enabled via shared `utils/modules/development`. - Host + guests apply the shared overlay `utils/overlays/packages.nix`. - `boot.kernelPackages` pins `linuxPackages_6_18` behind `lib.versionOlder pkgs.linux.version "6.18.22"`. - **Logging already runs on grafana-alloy** (promtail→alloy migration complete and live on all hosts). No logging work in this bump. **Desired behavior:** fw and all 5 guests evaluate and BUILD cleanly on nixos-26.05, services unchanged. Diff = channel flip + the forced fixes below. **Known 26.05 specifics for fw:** 1. **Insecure packages → `allowInsecurePredicate` (the real forced change).** On 26.05 the `makePythonWriter` interpreter guard force-evaluates the whole `pypy2Packages` set whenever any Rust package's cargo-vendor derivation is evaluated (pulled in via the users-groups shell-program-check). Its insecure members never enter the closure, but the check trips the insecure gate. Both hosts already on 26.05 (nb, nas) hit this and permit the family by prefix. Because **defining `allowInsecurePredicate` disables the `permittedInsecurePackages` list**, every existing list entry must move into the predicate, per nixpkgs instance: - **fw host:** predicate permitting `lib.hasPrefix "pypy2.7-"` **and** `openssl-1.1.1w`. - **web guest** (separate instance): predicate permitting `pypy2.7-` **and** `olm-3.2.16` (it builds `matrix-authentication-service`, which is Rust → trips the same guard). - **dev / fj-runner-1 / fj-runner-2 / openclaw:** add the `pypy2.7-` predicate to whichever instances eval/build flags (they carry no insecure list today). Use the predicate shape nb/nas use (read `pkg.name` with a `pname`-`version` fallback). Prefer the predicate over re-adding a static list — the list silently stops working once a predicate is defined. 2. **Drop the docker_29 pin.** On 26.05 the default docker is maintained (proven: nb runs 26.05 with docker via the development module and **no** pin). Remove the `dev` guest's `virtualisation.docker.package = pkgs.docker_29` and let the default apply. 3. **Do NOT touch pyload.** fw's pyload module is disabled (import commented out) and nothing forces `pkgs.pyload-ng`, so — unlike nas — fw needs **no** pyload-ng vendoring. The shared `utils/overlays/packages.nix` still wraps the (26.05-removed) `super.pyload-ng`, but it's lazy and never forced on fw; leave it alone. Editing the shared overlay would drag every host through the pre-commit dry-build. 4. **Kernel pin is fine.** `linuxPackages_6_18` is guarded by `versionOlder … "6.18.22"`; nas runs the identical pin on 26.05. Confirm it resolves/self-disables; don't remove it (separate cleanup). **Acceptance criteria:** - [ ] `hosts/fw/channel` → nixos-26.05. - [ ] Pre-commit eval green for fw AND every guest (`dev`, `web`, `fj-runner-1`, `fj-runner-2`, `openclaw`) on 26.05. - [ ] Build-verify: agent BUILDS (not just evals) the high-churn targets on the dev VM — the `web` guest's matrix stack (`matrix-synapse`, `matrix-authentication-service`, mautrix/olm), `docker`, and anything pulling a Rust cargo-vendor derivation — and reports any closure it could not build locally. Full host+guests build on fw is the maintainer's pre-merge gate (the dev VM can't hold the full keystone closure). *eval-only pre-commit misses 26.05 build-gates: udevadm verify, newly-broken pkgs.* - [ ] Host + each guest needing it use `allowInsecurePredicate` (`pypy2.7-` + the instance's existing entries); no leftover `permittedInsecurePackages` that the predicate silently disables. - [ ] docker_29 pin removed from the dev guest. - [ ] No logging/promtail/alloy changes; existing `utils/modules/alloy` imports untouched. - [ ] pyload untouched; `utils/overlays/packages.nix` untouched. - [ ] `system.stateVersion` stays 22.05; diff limited to channel flip + the fixes above. - [ ] Any new insecure/unfree allowance justified in the PR. **Out of scope:** - The promtail→alloy migration (done — #118/#124/#125). - The keystone reboot + runtime verification (DNS, Forgejo, all 5 guests, NIC drivers on 6.18) — that's #106 (`ready-for-human`). - Vendoring pyload / editing the shared overlay or shared utils modules. - Removing the `linuxPackages_6_18` pin (separate cleanup). - Other hosts' channels. **What changed vs the brief above:** ① removed the entire "drop promtail / set up alloy / no host imports alloy yet" workstream (done); ② added the `allowInsecurePredicate` + `pypy2.7-` migration with per-instance fold-in of `openssl-1.1.1w`/`olm-3.2.16`; ③ flipped pyload from "mirror nas" to "do NOT vendor"; ④ named all 5 guests (two runners); ⑤ made build-verify concrete.
Author
Owner

Recovered crashed AFK run → PR #128.

The AFK run for this bump crashed mid-flight. Its work was intact (uncommitted, staged + unstaged) in the afk/105 worktree — no commit had landed. Recovered it: backed up the diff, rebased onto current main (so it now includes #126), and committed.

Status — acceptance met:

  • Pre-commit eval green on all 6 hosts: fw + its 4 nixpkgs-tracking guests on 26.05; mail and web-arm on 25.11 (confirms the shared utils/modules/lego credentialsFileenvironmentFile change is backward-compatible); nas/nb/amzebs-01 unaffected.
  • Dry-run build of fw's full closure (host + all 5 guests) on 26.05 succeeds — fully instantiable; 1326 paths (≈3 GB) substitute from cache.
  • The two 26.05 build-only gates that slipped earlier bumps are N/A for fw: no custom udev rules (the %-escaping gate from nb #116 can't fire) and no allowBroken masking.

Residual: fw's pre-existing custom pins (invidious/Crystal, n8n/pnpm, mautrix-mattermost/Go, ai-mailer/Go) compile against the 26.05 toolchain for the first time at deploy — surfaced by the keystone reboot at the paired verify #106 (atomic switch protects runtime). A full local realize was skipped (eval-only bump convention; also this box has only ~15 GB free vs ~13 GB needed).

Ready for human land of #128.

**Recovered crashed AFK run → PR #128.** The AFK run for this bump crashed mid-flight. Its work was intact (uncommitted, staged + unstaged) in the `afk/105` worktree — no commit had landed. Recovered it: backed up the diff, rebased onto current `main` (so it now includes #126), and committed. **Status — acceptance met:** - **Pre-commit eval green on all 6 hosts**: fw + its 4 nixpkgs-tracking guests on 26.05; **mail** and **web-arm** on 25.11 (confirms the shared `utils/modules/lego` `credentialsFile`→`environmentFile` change is backward-compatible); nas/nb/amzebs-01 unaffected. - **Dry-run build of fw's full closure (host + all 5 guests) on 26.05 succeeds** — fully instantiable; 1326 paths (≈3 GB) substitute from cache. - The two 26.05 build-only gates that slipped earlier bumps are **N/A for fw**: no custom udev rules (the `%`-escaping gate from nb #116 can't fire) and no `allowBroken` masking. **Residual:** fw's pre-existing custom pins (invidious/Crystal, n8n/pnpm, mautrix-mattermost/Go, ai-mailer/Go) compile against the 26.05 toolchain for the first time at deploy — surfaced by the keystone reboot at the paired verify **#106** (atomic switch protects runtime). A full local realize was skipped (eval-only bump convention; also this box has only ~15 GB free vs ~13 GB needed). Ready for human land of #128.
Author
Owner

This was generated by AI during triage.

Delivered via PR #128 (merged): channel → nixos-26.05 for fw + its nixpkgs-tracking guests, permittedInsecurePackages → allowInsecurePredicate (host + web-02), docker_29 pin dropped, microvm /nix/store fsType set, cmp-spell unfree allowance, and lego credentialsFile → environmentFile. Paired verify #106 confirmed (fw rebooted onto 26.05). Closing the bump.

> *This was generated by AI during triage.* Delivered via PR #128 (merged): channel → nixos-26.05 for fw + its nixpkgs-tracking guests, `permittedInsecurePackages → allowInsecurePredicate` (host + web-02), `docker_29` pin dropped, microvm `/nix/store` fsType set, `cmp-spell` unfree allowance, and `lego` `credentialsFile → environmentFile`. Paired verify #106 confirmed (fw rebooted onto 26.05). Closing the bump.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Cloonar/nixos#105
No description provided.