Cloonar/nixos

Fork 0

feat(dev-new): move the WoW realm off the portable card #370

Merged

dominik.polakovics merged 2 commits from afk/wow-server-on-dev-new into main

2026-08-20 00:06:18 +02:00

dominik.polakovics commented

2026-08-19 23:45:48 +02:00

Owner

Moves the AzerothCore realm off the portable SD card onto dev-new, reachable as wow.cloonar.com from the LAN and over wg_cloonar. Design and reasoning: ADR-0027. Migration procedure: docs/runbooks/wow-realm-cutover.md.

Why

The card ran its own realm on whichever machine held it: server-up.sh copied the database off the card onto the internal drive, started three rootless podman containers on loopback, and server-down.sh copied the changed blocks back. All of that exists because the card travels — and it costs a sync protocol (sync tokens, dirty flag, libpod DB resets, orphan sweeps), podman on every play device, and a 200-bot worldserver running on the same machine that renders the game.

What this does

hosts/dev-new/modules/wow/ — native mysql84 with the four acore_* databases; authserver/worldserver as the existing prebuilt images from the Forgejo registry, on --network=host.
restartIfChanged = false on both container units plus an 05:00 restart timer. This host switches every five minutes; a channel bump moving the podman paths would otherwise disconnect a session mid-fight.
CPUWeight = 20 so the permanent bot population yields to builds and agents under contention, and only then.
services.mysqlBackup for acore_auth/acore_characters/acore_playerbots. acore_world is most of the bytes and is regenerated by wow-db-import.
wow.cloonar.com in fw's dnsmasq. Both realmlist.wtf and the realmd row use the name.
nb cleanup — wow-portable.nix drops podman, fuse-overlayfs and both user units; steam.nix drops the subuid/subgid FHS-sandbox workaround and WOW_SERVER_VIA_SYSTEMD. Only the card mount machinery stays, since the client still travels.

Two findings worth flagging

The card's AzerothCore checkout was dirty. Rev ceeb3116ebed carries an uncommitted 60-line fix to boss_valithria_dreamwalker.cpp that makes healing Valithria a valid pull, so a bot raid can do the encounter. It is compiled into the running worldserver and existed nowhere but that SD card. Committed here as patches/0001-valithria-heal-starts-the-encounter.patch.

The configs contained the DB password inline on five *DatabaseInfo lines. Committed scrubbed (__set_via_AC_env__); the real credentials come from sops via AC_*_DATABASE_INFO, which override the file — a mechanism the current setup already proves (Server.log: Found config value 'PlayerbotsDatabaseInfo' from environment variable).

Not done, deliberately

session.sh is untouched. Editing it now would point the working setup at a server that does not exist. It is step 6 of the runbook.
Offsite backups are not wired. dev-new needs a Hetzner Storage Box sub-account that does not exist yet; mysqlBackup writes daily dumps to /var/backup/mysql locally in the meantime. The nix snippet is in the runbook prerequisites.

Before merging

hosts/dev-new/secrets.sops.yaml needs wow-db-password (alphanumeric — it is interpolated into SQL by wow-provision.service), and the images must be pushed to git.cloonar.com/cloonar as :ceeb3116.

Ordering trap: the mysqldump and podman save in runbook phase 2 must happen before this deploys, because the deploy removes podman from nb.

Verification

scripts/test-configuration dry-builds clean for dev-new, fw and nb. Nothing is deployed and no data has moved; the card's realm is untouched and still authoritative.

Related: #369 (put the card's remaining scripts under version control).

Moves the AzerothCore realm off the portable SD card onto `dev-new`, reachable as `wow.cloonar.com` from the LAN and over `wg_cloonar`. Design and reasoning: **ADR-0027**. Migration procedure: **`docs/runbooks/wow-realm-cutover.md`**. ## Why The card ran its own realm on whichever machine held it: `server-up.sh` copied the database off the card onto the internal drive, started three rootless podman containers on loopback, and `server-down.sh` copied the changed blocks back. All of that exists because the card travels — and it costs a sync protocol (sync tokens, dirty flag, libpod DB resets, orphan sweeps), podman on every play device, and a 200-bot worldserver running on the same machine that renders the game. ## What this does - **`hosts/dev-new/modules/wow/`** — native `mysql84` with the four `acore_*` databases; `authserver`/`worldserver` as the existing prebuilt images from the Forgejo registry, on `--network=host`. - **`restartIfChanged = false`** on both container units plus an 05:00 restart timer. This host switches every five minutes; a channel bump moving the podman paths would otherwise disconnect a session mid-fight. - **`CPUWeight = 20`** so the permanent bot population yields to builds and agents under contention, and only then. - **`services.mysqlBackup`** for `acore_auth`/`acore_characters`/`acore_playerbots`. `acore_world` is most of the bytes and is regenerated by `wow-db-import`. - **`wow.cloonar.com`** in fw's dnsmasq. Both `realmlist.wtf` and the `realmd` row use the name. - **nb cleanup** — `wow-portable.nix` drops podman, fuse-overlayfs and both user units; `steam.nix` drops the subuid/subgid FHS-sandbox workaround and `WOW_SERVER_VIA_SYSTEMD`. Only the card mount machinery stays, since the client still travels. ## Two findings worth flagging **The card's AzerothCore checkout was dirty.** Rev `ceeb3116ebed` carries an uncommitted 60-line fix to `boss_valithria_dreamwalker.cpp` that makes healing Valithria a valid pull, so a bot raid can do the encounter. It is compiled into the running worldserver and existed nowhere but that SD card. Committed here as `patches/0001-valithria-heal-starts-the-encounter.patch`. **The configs contained the DB password inline** on five `*DatabaseInfo` lines. Committed scrubbed (`__set_via_AC_env__`); the real credentials come from sops via `AC_*_DATABASE_INFO`, which override the file — a mechanism the current setup already proves (`Server.log`: *Found config value 'PlayerbotsDatabaseInfo' from environment variable*). ## Not done, deliberately - **`session.sh` is untouched.** Editing it now would point the working setup at a server that does not exist. It is step 6 of the runbook. - **Offsite backups are not wired.** `dev-new` needs a Hetzner Storage Box sub-account that does not exist yet; `mysqlBackup` writes daily dumps to `/var/backup/mysql` locally in the meantime. The nix snippet is in the runbook prerequisites. ## Before merging `hosts/dev-new/secrets.sops.yaml` needs `wow-db-password` (alphanumeric — it is interpolated into SQL by `wow-provision.service`), and the images must be pushed to `git.cloonar.com/cloonar` as `:ceeb3116`. **Ordering trap:** the `mysqldump` and `podman save` in runbook phase 2 must happen *before* this deploys, because the deploy removes podman from nb. ## Verification `scripts/test-configuration` dry-builds clean for `dev-new`, `fw` and `nb`. Nothing is deployed and no data has moved; the card's realm is untouched and still authoritative. Related: #369 (put the card's remaining scripts under version control).

dominik.polakovics added 1 commit

2026-08-19 23:45:48 +02:00

feat(dev-new): move the WoW realm off the portable card 8c184775c5

The AzerothCore realm ran on whichever machine held the SD card, with the
database copied on and off it every session. It now runs on dev-new and every
device joins as wow.cloonar.com over the LAN or wg_cloonar.

MySQL is native (mysql84, declarative databases, mysqlBackup); authserver and
worldserver stay as the prebuilt images, pulled from the Forgejo registry and
run with --network=host so MySQL never leaves 127.0.0.1. Deploys never restart
the world - it cycles at 05:00 instead. See ADR-0027 and the cutover runbook.

nb loses podman, fuse-overlayfs, both user units and the Steam FHS subuid
workaround; only the card mount stays. The card's local Valithria fix, which
existed nowhere but that SD card, is committed as a patch.

dominik.polakovics added 1 commit

2026-08-20 00:01:56 +02:00

fix(dev-new): digest-pin the acore images, correct the cutover runbook 5ff283b80e

The images are pushed and pull anonymously from the Forgejo registry, so pin
them by digest as well as tag (ADR-0026) - a re-tag can no longer change what
the host runs.

Runbook corrections from actually running phases 1-4: every podman command on
the card needs `source bin/common.sh; setup_podman_env`, or it silently finds
no containers rather than erroring; the servers are stopped before the dump,
since 200 bots write continuously and --single-transaction does not cover
MyISAM; and the artefacts go to /home rather than nb's 16 GB tmpfs root.

dominik.polakovics commented

2026-08-20 00:04:42 +02:00

Author

Owner

Cutover phases 1–4 are done, so this PR is unblocked. Nothing is deployed and the card is still authoritative — a rollback needs no action beyond not merging.

Captured from the card (/home/dominik/wow-cutover/ on nb):

acore-precutover.sql — 363 MB, all four databases, verified complete. Taken with the world and auth servers stopped so characters were flushed and nothing was mid-write.
acore-{worldserver,authserver,db-import}.tar — 712 MB / 231 MB / 1.2 GB.

The card stack was then stopped cleanly; server-down.sh copied the database back (41 MB newly written).

Staged on dev-new: /var/lib/wow/data (3.1 GB, dbc/maps/mmaps/vmaps), /var/lib/wow/modules (108 MB), and the dump at /var/lib/wow/acore-precutover.sql.

Images pushed to git.cloonar.com/cloonar/acore-*:ceeb3116 and confirmed to pull anonymously from dev-new — no registry credentials are needed on the host. Now digest-pinned in the module as well as tagged, per ADR-0026:

authserver   sha256:9408d8ae6db1aaa06ad2ab0547e7e85a80509de2c399e1ac7a6b45c7432868ac
worldserver  sha256:199273099f808158c113856685eec9b508392a1f0bf4d0db3bf8778de2864c7a
db-import    sha256:fe072afc95c688d10321a68702b3a1ee6f29790fb30e776203f6aae9e3f335d8

wow-db-password is in hosts/dev-new/secrets.sops.yaml.

Runbook corrections, from actually running it

Every podman command against the card needs source bin/common.sh; setup_podman_env first. Without it podman uses its default store and reports no containers at all rather than an error — which reads as "the stack isn't running" when it is.
The dump now stops acore-worldserver and acore-authserver first. 200 bots write continuously, and --single-transaction does not cover MyISAM tables.
Artefacts go to /home, not /tmp — nb's root is a 16 GB tmpfs and the pre-cutover dump has to survive a reboot.

Cleanup needed (my mistake, needs the web UI)

Three junk container repos exist on fw from a first push where zsh's $i:c parameter modifier ate the tag separator: acore-worldservereeb3116, acore-authservereeb3116, acore-db-importeeb3116, about 2.1 GB total. tea cannot delete packages, so they need removing from the Forgejo packages UI under the cloonar org.

After merge

Runbook phase 5 (import the dump, run wow-db-import, start the servers, verify the realm row and that 3724/8085 listen), then phase 6 (session.sh: realmlist to wow.cloonar.com, drop the server bring-up calls).

Still open, unchanged: offsite backups need a Hetzner Storage Box sub-account, and the world tick at 500–1000 bots is unmeasured by design.

Cutover phases 1–4 are done, so this PR is unblocked. Nothing is deployed and **the card is still authoritative** — a rollback needs no action beyond not merging. **Captured from the card** (`/home/dominik/wow-cutover/` on nb): - `acore-precutover.sql` — 363 MB, all four databases, verified complete. Taken with the world and auth servers stopped so characters were flushed and nothing was mid-write. - `acore-{worldserver,authserver,db-import}.tar` — 712 MB / 231 MB / 1.2 GB. The card stack was then stopped cleanly; `server-down.sh` copied the database back (41 MB newly written). **Staged on dev-new**: `/var/lib/wow/data` (3.1 GB, dbc/maps/mmaps/vmaps), `/var/lib/wow/modules` (108 MB), and the dump at `/var/lib/wow/acore-precutover.sql`. **Images pushed** to `git.cloonar.com/cloonar/acore-*:ceeb3116` and confirmed to **pull anonymously from dev-new** — no registry credentials are needed on the host. Now digest-pinned in the module as well as tagged, per ADR-0026: ``` authserver sha256:9408d8ae6db1aaa06ad2ab0547e7e85a80509de2c399e1ac7a6b45c7432868ac worldserver sha256:199273099f808158c113856685eec9b508392a1f0bf4d0db3bf8778de2864c7a db-import sha256:fe072afc95c688d10321a68702b3a1ee6f29790fb30e776203f6aae9e3f335d8 ``` **`wow-db-password` is in `hosts/dev-new/secrets.sops.yaml`.** ## Runbook corrections, from actually running it - Every podman command against the card needs `source bin/common.sh; setup_podman_env` first. Without it podman uses its default store and reports **no containers at all** rather than an error — which reads as "the stack isn't running" when it is. - The dump now stops `acore-worldserver` and `acore-authserver` first. 200 bots write continuously, and `--single-transaction` does not cover MyISAM tables. - Artefacts go to `/home`, not `/tmp` — nb's root is a 16 GB tmpfs and the pre-cutover dump has to survive a reboot. ## Cleanup needed (my mistake, needs the web UI) Three junk container repos exist on fw from a first push where zsh's `$i:c` parameter modifier ate the tag separator: `acore-worldservereeb3116`, `acore-authservereeb3116`, `acore-db-importeeb3116`, about 2.1 GB total. `tea` cannot delete packages, so they need removing from the Forgejo packages UI under the `cloonar` org. ## After merge Runbook phase 5 (import the dump, run `wow-db-import`, start the servers, verify the realm row and that 3724/8085 listen), then phase 6 (`session.sh`: realmlist to `wow.cloonar.com`, drop the server bring-up calls). Still open, unchanged: offsite backups need a Hetzner Storage Box sub-account, and the world tick at 500–1000 bots is unmeasured by design.

dominik.polakovics merged commit dddd9c0035 into main

2026-08-20 00:06:18 +02:00

dominik.polakovics deleted branch afk/wow-server-on-dev-new

2026-08-20 00:06:19 +02:00

dominik.polakovics referenced this pull request from a commit

2026-08-20 00:06:20 +02:00

Merge pull request 'feat(dev-new): move the WoW realm off the portable card' (#370) from afk/wow-server-on-dev-new into main