fix(fw): curb disk growth (14d gc retention, journald cap, qemu discard) #193

Open

dominik.polakovics wants to merge 1 commit from fix/fw-disk-reclaim into main

dominik.polakovics commented

2026-06-28 14:17:48 +02:00

Owner

Why

fw root (/dev/nvme0n1p2, 468G) was at 85% — 377G used, 68G free. No single leak; the disk is dominated by VM images (~227G) and the nix store (93G). These three changes slow the recurring fill (the large one-time reclaim — orphaned 36G microvm-dev image, retired-container data, qcow2 compaction — is operational and done by hand on the host).

What

nix.gc 60d → 14d retention. The 60-day window pinned ~2769 system generations and kept /nix/store at ~93G; the weekly gc was only freeing ~1.3G/run because almost everything was inside the window. 14d keeps ample rollback while letting gc actually reclaim.
journald cap SystemMaxUse=500M. Archived journals had grown to ~4G. Alloy ships logs off-box, so a small on-host ring buffer is enough.
qemu-vm: discard=unmap,detect-zeroes=unmap on the disk drive. This is the root cause of qcow2 bloat — without it, freed guest blocks never return to the host and the image only ever grows to its high-water mark (dev-vm 95G/100G, openclaw-vm 34G). With discard, guest fstrim/-o discard shrinks the qcow2. Takes effect on the next deliberate systemctl restart <name>-vm (units are restartIfChanged = false).

Test

Pre-commit dry-build passed for all hosts (amzebs-01, dev, fw, mail, nas, nb, web-arm).

## Why fw root (`/dev/nvme0n1p2`, 468G) was at **85% — 377G used, 68G free**. No single leak; the disk is dominated by VM images (~227G) and the nix store (93G). These three changes slow the recurring fill (the large one-time reclaim — orphaned 36G microvm-dev image, retired-container data, qcow2 compaction — is operational and done by hand on the host). ## What - **`nix.gc` 60d → 14d retention.** The 60-day window pinned ~2769 system generations and kept `/nix/store` at ~93G; the weekly gc was only freeing ~1.3G/run because almost everything was inside the window. 14d keeps ample rollback while letting gc actually reclaim. - **journald cap `SystemMaxUse=500M`.** Archived journals had grown to ~4G. Alloy ships logs off-box, so a small on-host ring buffer is enough. - **qemu-vm: `discard=unmap,detect-zeroes=unmap` on the disk drive.** This is the root cause of qcow2 bloat — without it, freed guest blocks never return to the host and the image only ever grows to its high-water mark (dev-vm 95G/100G, openclaw-vm 34G). With discard, guest `fstrim`/`-o discard` shrinks the qcow2. Takes effect on the next deliberate `systemctl restart <name>-vm` (units are `restartIfChanged = false`). ## Test Pre-commit dry-build passed for all hosts (amzebs-01, dev, fw, mail, nas, nb, web-arm).

dominik.polakovics added 1 commit

2026-06-28 14:17:48 +02:00

fix(fw): curb disk growth (14d gc retention, journald cap, qemu discard) 9687d42ebc

fw root was at 85% (377G/468G). Three changes to slow the fill:

- nix.gc: 60d -> 14d retention. The 60d window pinned ~2769 system
  generations and kept /nix/store at ~93G; the weekly gc freed only
  ~1.3G/run. 14d keeps ample rollback while letting gc actually reclaim.
- journald: cap SystemMaxUse=500M (archived logs had grown to ~4G; Alloy
  ships logs off-box, so a small local ring buffer suffices).
- qemu-vm: add discard=unmap,detect-zeroes=unmap to the disk drive so
  guests can hand freed blocks back to the host and the qcow2 shrinks
  instead of only ever growing to its high-water mark. Latent until each
  VM is restarted (units have restartIfChanged=false).

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/fw-disk-reclaim:fix/fw-disk-reclaim

git switch fix/fw-disk-reclaim

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff fix/fw-disk-reclaim

git switch fix/fw-disk-reclaim

git rebase main

git switch main

git merge --ff-only fix/fw-disk-reclaim

git switch fix/fw-disk-reclaim

git rebase main

git switch main

git merge --no-ff fix/fw-disk-reclaim

git switch main

git merge --squash fix/fw-disk-reclaim