Auto-update invidious-companion (+ decide Invidious) and harden companion env-file generation #89
Labels
No labels
bug
enhancement
in-progress
needs-info
needs-triage
p0
ready-for-agent
ready-for-human
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Cloonar/nixos#89
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Background
Diagnosed 2026-06-04: Yattee couldn't play videos;
invidious-companionhad been failing PO-token validation because its:latestimage was frozen at a 2026-03-04 build for ~3 months.--pull=neweronly re-pulls on container (re)creation, and nothing ever triggered that (no reboot / config change / timer). Manually pulling the current build (2026-06-03) + restarting restored playback (verified:/api/v1/videos/…returnsadaptiveFormats, no error). That fix is runtime-only / uncommitted — this issue makes it permanent.File:
hosts/fw/vms/web/invidious.nix. Related: #88 (agent SSH access to the VMs).1. Auto-update the companion container
Replace the ineffective
--pull=newerwith real auto-update:io.containers.autoupdate=registrytovirtualisation.oci-containers.containers.invidious-companion.podman-auto-update.timer→ runspodman auto-updatedaily). Note: there is no singlevirtualisation.podman.autoUpdate.enableoption — wire the systemd timer/service (or enable podman's packagedpodman-auto-update.timer).virtualisation.oci-containers.containers.invidious-companion.podman.sdnotify = "healthy"+ a container healthcheck, sopodman auto-update --rollbackreverts automatically if a new:latestis broken.2. (REQUIRED — must land with #1) Harden the companion env file
Today the env file (
PORT/HOST/SERVER_SECRET_KEY) is written to volatile/run/invidious-companion/envby a boot-onlyoneshot+RemainAfterExit=truegenerator. During the last uptime the entire/run/invidious-companion/directory vanished, so the restart failed with:(Had to recreate the dir + env by hand.) This must be fixed before/with auto-update:
podman auto-updaterestarts the container on every pull, so without this fix every auto-update would brick companion exactly as observed.Fix: regenerate the env on every companion start (drop
RemainAfterExit, or move generation into the container unit'sExecStartPre/preStart) and back it with systemdRuntimeDirectory=invidious-companion(+RuntimeDirectoryPreserve) instead of a tmpfiles dir in/run. Then no restart can fail on a missing env file regardless of what cleans/run. (Exact cause of the dir's disappearance unconfirmed — likely a tmpfiles--remove/clean orRuntimeDirectoryteardown — but the fix is robust either way.)3. Auto-update Invidious itself — needs a decision
Invidious is not a container here — it's the native
services.invidiousnixpkgs module (which also wireshttp3-ytproxy, the nginx vhost, the DB, and admin-user init). So container auto-update doesn't apply. Options:channelfile is bumped + bento rebuild. Lowest risk, keeps module integration, but updates are manual and may lag upstream (and Invidious, like companion, needs frequent updates to track YouTube).quay.io/invidious/invidious:latest) for the same auto-update treatment as companion. True parity + fastest YouTube-tracking, but a significant migration: reimplement DB/nginx/TLS/http3-ytproxy/admin-init/companion wiring that the module does today, plus Postgres data migration.Recommendation: ship #1+#2 now (clear, low-risk win); decide #3 separately — likely (a) + a discipline of regular channel bumps, unless we commit to the (b) migration.
Acceptance
Agent Brief
Category: enhancement
Summary: Make
invidious-companionauto-update its container image and regenerate its env file on every start; record the decision to keep Invidious itself native.Current behavior:
virtualisation.oci-containers.containers.invidious-companioncontainer (imagequay.io/invidious/invidious-companion:latest) relies on--pull=newer, which only re-pulls on container (re)creation. Nothing triggers re-creation, so the image froze at a ~3-month-old build and broke PO-token validation / playback. Fixed manually at runtime; uncommitted.PORT/HOST/SERVER_SECRET_KEY) is written to/run/invidious-companion/envby aoneshot+RemainAfterExit=truegenerator (systemd.services.invidious-companion-env-generate) that runs once at boot and never again, backed by a tmpfiles/rundirectory. When/run/invidious-companion/was cleaned, the env file vanished and the container failed to start (open /run/invidious-companion/env: no such file or directory), needing manual recreation.services.invidiousnixpkgs module (not a container).Desired behavior:
podman auto-updateshould roll back to the last working image./runcleanup, restart, reboot, or auto-update pull can leave the container without it. Generation must no longer be a once-per-boot,RemainAfterExit-latched step.Key interfaces / config shapes (durable Nix attributes):
virtualisation.oci-containers.containers.invidious-companion: add theio.containers.autoupdate=registrylabel. Recommended: set.podman.sdnotify = "healthy"with a container healthcheck so rollback triggers on a functionally-broken image — but verify the companion image exposes a real health endpoint on :8282 first; if not, add a--health-cmdviaextraOptions, or fall back to default sdnotify (rollback then only covers hard start failures). Do not invent a health endpoint.virtualisation.podman.autoUpdateNixOS option (verified on nixos-25.11). Wire it whichever way dry-builds cleanly: enable podman's packagedpodman-auto-update.timer, or define a smalloneshotservice runningpodman auto-update+ a dailysystemd.timer(OnCalendar=daily,Persistent=true).RemainAfterExit=true(or move generation into the companion unit'sExecStartPre/preStart) so it runs on every start, and back the directory with systemdRuntimeDirectory=invidious-companion(+RuntimeDirectoryPreserve) on the owning unit instead of the tmpfiles/run/invidious-companionrule, tying the dir's lifecycle to the service.invidious-companion-keySOPS secret — don't change secret handling.quay.io/invidious/invidious:latest) and its cost (reimplementing DB/nginx/TLS/http3-ytproxy/admin-init wiring + Postgres data migration).Acceptance criteria:
io.containers.autoupdate=registryand a recurringpodman auto-updateis armed (timer enabled/active).RemainAfterExit=trueas a once-per-boot latch; the env is regenerated on every start and its directory is provided byRuntimeDirectory(no/runtmpfiles rule for it)./run-wipe + restart cycle with no manual env recreation. (Runtime check — see verification note.)Out of scope:
--cap-drop=ALL,no-new-privileges,--read-only) or itsinvidious-netnetwork.companion-config.jsongenerator, the SOPS secret, and any nginx/http3-ytproxy/CORS config — only the companion env-file generator is in scope.Verification note (ties to #88): This module lives in the fw web microVM, which agents currently can't SSH into (#88). Make the change and rely on the pre-commit dry-build as the gate; the runtime acceptance checks (survives
/run-wipe restart without manual env recreation; timer pulls a newer digest over time) must be confirmed by a human/HITL on the VM after deploy. A dry-build-passing PR is the agent's completion bar; runtime sign-off is the maintainer's.