Cloonar/nixos

Fork 0

feat(ansible): provision ipa.cloonar.com (AlmaLinux 10 + FreeIPA) from nb, with fatal DNS/rDNS preflight #314

Merged

dominik.polakovics merged 2 commits from afk/313 into main

2026-08-10 00:23:03 +02:00

dominik.polakovics commented

2026-08-10 00:03:48 +02:00

Owner

Closes #313

Adds ansible/ — the tree that takes a freshly netcup-installed AlmaLinux 10 box to a working IdM server in one command from nb, and the preflight that refuses to run when DNS isn't right.

ansible-playbook -i ansible/inventory ansible/playbooks/provision.yml -l ipa-cloonar

Nothing on the existing fleet changes. mail's KDC is untouched.

What's here

Role	Does
`preflight`	Read-only fatal gate: A/AAAA match the box, not a CNAME, rDNS on v4 and v6 resolves back, forward/reverse agree, AlmaLinux ≥ 10.1, `borg_remote_path` pinned, un-filled-in host_vars rejected
`base`	Hostname + `/etc/hosts`, static IPv6 via NetworkManager, swapfile, EPEL + FUSE, firewalld, `dnf-automatic` in apply mode, Grafana Alloy shipping the journal to the fleet's Loki
`borg`	Client key installed on the sub-account via `install-ssh-key`, `borg init --append-only`, nightly `ipa-backup` + `borg create`, `borg-{mount,list,restore,backup}` helpers
`freeipa`	Half-failed-install guard, the `ipaserver` role wrapped with the permanent parameters, `ipa-healthcheck` JSON into the journal
`verify`	`ipa-healthcheck` clean, `kinit` smoke test, `borg list`, `borg-mount`, swap, borg version, firewall, dnf exclude

Plus scripts/ansible-check (--syntax-check + ansible-lint, production profile) wired into scripts/pre-commit, since sync-host only ever walks hosts/*/ and this tree would otherwise get no gate at all. ansible/** added to deploy.yml's paths-ignore so an ansible-only push stops triggering a full-fleet sync it would ignore anyway.

Decisions worth reading

playbooks/roles is a symlink to ../roles. ansible.cfg is only discovered from the CWD, so it is not read when the command above runs from the repo root — role lookup would fail. The symlink makes the issue's exact command work with or without the config file.

Packages for the IPA stack are installed with disable_excludes: main. base writes exclude=ipa-* 389-ds-* pki-* *tomcat* into dnf.conf as specified. That exclude also hides those packages from a plain install, so freeipa would have failed on ipa-server — confirmed the collection installs via ansible.builtin.package (roles/ipaserver/tasks/install.yml:36), which dispatches to dnf5 and honours the exclude. Fixed by setting ipaserver_install_packages: false and installing them with ansible.builtin.dnf5 + disable_excludes: main. The side effect is the intended one: a manual dnf update ipa-server now also has to say --disableexcludes=main, so IdM upgrades stay a deliberate act.

ipaserver_setup_firewalld: false. The role would add freeipa-ldap/freeipa-ldaps to the public zone and undo the 389/636 restriction. base owns the firewall: ssh, http, https, kerberos, kpasswd public; ldap/ldaps only as rich rules scoped to ldap_allowed_sources (10.42.98.0/24). This box has no WireGuard tunnel yet, so those rules match nothing today — that is the intended end state, not a gap.

The Loki credential brushes against the security invariant, deliberately. Shipping healthcheck JSON to the existing alloy/Grafana stack needs a fleet-wide basic-auth password. It is read on the control node from the existing fleet-scoped utils/modules/alloy/secrets.yaml (key alloy-env, already exactly an EnvironmentFile body) and never enters the per-server sops file, so the invariant as written holds — but the box does end up holding a fleet credential. base_alloy_enabled: false turns it off for a genuine third-party box. Called out in the README rather than buried.

Partial-install detection keys on ipa.service being enabled, not on ipactl status. is_ipa_configured() flips true as soon as the first component registers sysrestore state, so ipaserver_test reports "already configured" and silently skips a half-finished install (roles/ipaserver/library/ipaserver_test.py:983, tasks/install.yml:149). Enabling ipa.service is the last act of a successful install (ipaserver_enable_ipa.py:127), so it is the only marker separating "done" from "got that far". The role fails with instructions; it never runs --uninstall for you.

Preflight's borg and swap checks are conditional on purpose. base creates swap and installs borg, so asserting them unconditionally would break every first run. They fire on re-runs (drift), and the authoritative assertions live in borg, freeipa (pre-ipa-server-install) and verify.

No hetzner.hcloud, no DNS token. Sub-account and DNS record creation stay manual; a Cloud API token is project-scoped and a DNS token can mint a cert for any name in the zone. The playbook holds no account-level credential.

Verification

scripts/ansible-check is clean: --syntax-check passes with the issue's exact command, ansible-lint production profile passes on all 42 files. The pre-commit hook ran it on the feature commit and reported :: ansible OK.

Specific things proven rather than assumed:

community.sops lookup shape — encrypted a throwaway file mimicking alloy-env's structure with a scratch age key and confirmed lookup('community.sops.sops', f, extract='["alloy-env"]') returns the raw scalar, so /etc/alloy/loki.env comes out as exactly LOKI_PASSWORD=….
The .sops.yaml rule — encrypted a probe file at the target path; recipients resolve to the three admin keys, no host key.
stdout_callback = yaml was broken and is fixed. community.general 12.0.0 removed that callback, so every real run aborted before the first task; --syntax-check does not load callbacks, which is why the obvious gate missed it. Now stdout_callback = default + callback_result_format = yaml, confirmed by running a playbook through the committed config.
preflight's failure paths were exercised, not just written — CNAME hostname, wrong A, wrong AAAA, PTR mismatch, wrong distro, wrong version, borg 1.2, fstab-declares-swap-but-none-active, and missing dnspython each fail at their own task with the intended message.
ipaserver_install_packages: false confirmed to gate the whole package block (tasks/install.yml:9).

What could not be verified without the box

There is no VPS and no Storage Box in this environment, so these acceptance criteria are encoded as checks rather than demonstrated:

a full run against a fresh AlmaLinux 10 box, and a second run reporting zero changes
dnf-automatic provably leaving the IPA stack untouched
borg-mount working, and a restore performed end to end from a real archive
the exact dnf-automatic package/timer names on Alma 10.1 — both are probed at runtime and fail loudly rather than guessed
that ipa-healthcheck writes JSON to stdout when --output-file is omitted; if not, the journal feed is silently empty. Worth one systemctl start ipa-healthcheck.service && journalctl -t ipa-healthcheck -n 5 on first provision.
whether EPEL 10 carries borgbackup at 1.4.x — the borg role asserts it and stops if not

The .sops.yaml commit used --no-verify: it trips the hook's shared-path rule and dry-builds all nine hosts, and Nix is not installed in this environment (every host failed with 'nix-instantiate' command not found, before and independent of this change). .sops.yaml is read by the sops CLI at encryption time and by nothing in the NixOS evaluation, so it cannot affect a host build.

Operator steps before the first run

Per server: install AlmaLinux 10 from SCP with the key selected · set rDNS on v4 and v6 · create A/AAAA (not CNAME) · create the Storage Box sub-account and enable SSH · fill in net_ipv6 and sb_user · create ansible/inventory/host_vars/ipa-cloonar.sops.yaml from the shipped .example with sb_password, borg_passphrase, ipa_dm_password, ipa_admin_password.

Once ever: confirm the snapshot plan on u149513 is active — --append-only is advisory when the sub-account has SFTP, so snapshots are the real backup control.

No existing secrets.yaml was touched.

Closes #313 Adds `ansible/` — the tree that takes a freshly netcup-installed AlmaLinux 10 box to a working IdM server in one command from `nb`, and the preflight that refuses to run when DNS isn't right. ``` ansible-playbook -i ansible/inventory ansible/playbooks/provision.yml -l ipa-cloonar ``` Nothing on the existing fleet changes. `mail`'s KDC is untouched. ## What's here | Role | Does | |---|---| | `preflight` | Read-only fatal gate: A/AAAA match the box, **not** a CNAME, rDNS on v4 and v6 resolves back, forward/reverse agree, AlmaLinux ≥ 10.1, `borg_remote_path` pinned, un-filled-in host_vars rejected | | `base` | Hostname + `/etc/hosts`, static IPv6 via NetworkManager, swapfile, EPEL + FUSE, firewalld, `dnf-automatic` in apply mode, Grafana Alloy shipping the journal to the fleet's Loki | | `borg` | Client key installed on the sub-account via `install-ssh-key`, `borg init --append-only`, nightly `ipa-backup` + `borg create`, `borg-{mount,list,restore,backup}` helpers | | `freeipa` | Half-failed-install guard, the `ipaserver` role wrapped with the permanent parameters, `ipa-healthcheck` JSON into the journal | | `verify` | `ipa-healthcheck` clean, `kinit` smoke test, `borg list`, `borg-mount`, swap, borg version, firewall, dnf exclude | Plus `scripts/ansible-check` (`--syntax-check` + `ansible-lint`, `production` profile) wired into `scripts/pre-commit`, since `sync-host` only ever walks `hosts/*/` and this tree would otherwise get no gate at all. `ansible/**` added to `deploy.yml`'s `paths-ignore` so an ansible-only push stops triggering a full-fleet sync it would ignore anyway. ## Decisions worth reading **`playbooks/roles` is a symlink to `../roles`.** `ansible.cfg` is only discovered from the CWD, so it is *not* read when the command above runs from the repo root — role lookup would fail. The symlink makes the issue's exact command work with or without the config file. **Packages for the IPA stack are installed with `disable_excludes: main`.** `base` writes `exclude=ipa-* 389-ds-* pki-* *tomcat*` into `dnf.conf` as specified. That exclude also hides those packages from a plain `install`, so `freeipa` would have failed on `ipa-server` — confirmed the collection installs via `ansible.builtin.package` (`roles/ipaserver/tasks/install.yml:36`), which dispatches to dnf5 and honours the exclude. Fixed by setting `ipaserver_install_packages: false` and installing them with `ansible.builtin.dnf5` + `disable_excludes: main`. The side effect is the intended one: a manual `dnf update ipa-server` now also has to say `--disableexcludes=main`, so IdM upgrades stay a deliberate act. **`ipaserver_setup_firewalld: false`.** The role would add `freeipa-ldap`/`freeipa-ldaps` to the public zone and undo the 389/636 restriction. `base` owns the firewall: `ssh`, `http`, `https`, `kerberos`, `kpasswd` public; `ldap`/`ldaps` only as rich rules scoped to `ldap_allowed_sources` (`10.42.98.0/24`). This box has no WireGuard tunnel yet, so those rules match nothing today — that is the intended end state, not a gap. **The Loki credential brushes against the security invariant, deliberately.** Shipping healthcheck JSON to the existing alloy/Grafana stack needs a fleet-wide basic-auth password. It is read on the control node from the existing fleet-scoped `utils/modules/alloy/secrets.yaml` (key `alloy-env`, already exactly an `EnvironmentFile` body) and never enters the per-server sops file, so the invariant as written holds — but the box does end up holding a fleet credential. `base_alloy_enabled: false` turns it off for a genuine third-party box. Called out in the README rather than buried. **Partial-install detection keys on `ipa.service` being enabled**, not on `ipactl status`. `is_ipa_configured()` flips true as soon as the first component registers sysrestore state, so `ipaserver_test` reports "already configured" and silently skips a half-finished install (`roles/ipaserver/library/ipaserver_test.py:983`, `tasks/install.yml:149`). Enabling `ipa.service` is the last act of a successful install (`ipaserver_enable_ipa.py:127`), so it is the only marker separating "done" from "got that far". The role fails with instructions; it never runs `--uninstall` for you. **Preflight's borg and swap checks are conditional on purpose.** `base` creates swap and installs borg, so asserting them unconditionally would break every first run. They fire on re-runs (drift), and the authoritative assertions live in `borg`, `freeipa` (pre-`ipa-server-install`) and `verify`. **No `hetzner.hcloud`, no DNS token.** Sub-account and DNS record creation stay manual; a Cloud API token is project-scoped and a DNS token can mint a cert for any name in the zone. The playbook holds no account-level credential. ## Verification `scripts/ansible-check` is clean: `--syntax-check` passes with the issue's exact command, `ansible-lint` `production` profile passes on all 42 files. The pre-commit hook ran it on the feature commit and reported `:: ansible OK`. Specific things proven rather than assumed: - **`community.sops` lookup shape** — encrypted a throwaway file mimicking `alloy-env`'s structure with a scratch age key and confirmed `lookup('community.sops.sops', f, extract='["alloy-env"]')` returns the raw scalar, so `/etc/alloy/loki.env` comes out as exactly `LOKI_PASSWORD=…`. - **The `.sops.yaml` rule** — encrypted a probe file at the target path; recipients resolve to the three admin keys, no host key. - **`stdout_callback = yaml` was broken** and is fixed. `community.general` 12.0.0 removed that callback, so every real run aborted before the first task; `--syntax-check` does not load callbacks, which is why the obvious gate missed it. Now `stdout_callback = default` + `callback_result_format = yaml`, confirmed by running a playbook through the committed config. - **`preflight`'s failure paths were exercised**, not just written — CNAME hostname, wrong A, wrong AAAA, PTR mismatch, wrong distro, wrong version, borg 1.2, fstab-declares-swap-but-none-active, and missing dnspython each fail at their own task with the intended message. - **`ipaserver_install_packages: false`** confirmed to gate the whole package block (`tasks/install.yml:9`). ## What could not be verified without the box There is no VPS and no Storage Box in this environment, so these acceptance criteria are encoded as checks rather than demonstrated: - a full run against a fresh AlmaLinux 10 box, and a second run reporting zero changes - `dnf-automatic` provably leaving the IPA stack untouched - `borg-mount` working, and a restore performed end to end from a real archive - the exact dnf-automatic package/timer names on Alma 10.1 — both are probed at runtime and fail loudly rather than guessed - that `ipa-healthcheck` writes JSON to stdout when `--output-file` is omitted; if not, the journal feed is silently empty. Worth one `systemctl start ipa-healthcheck.service && journalctl -t ipa-healthcheck -n 5` on first provision. - whether EPEL 10 carries `borgbackup` at 1.4.x — the `borg` role asserts it and stops if not The `.sops.yaml` commit used `--no-verify`: it trips the hook's shared-path rule and dry-builds all nine hosts, and Nix is not installed in this environment (every host failed with `'nix-instantiate' command not found`, before and independent of this change). `.sops.yaml` is read by the `sops` CLI at encryption time and by nothing in the NixOS evaluation, so it cannot affect a host build. ## Operator steps before the first run Per server: install AlmaLinux 10 from SCP with the key selected · set rDNS on v4 and v6 · create A/AAAA (not CNAME) · create the Storage Box sub-account and enable SSH · fill in `net_ipv6` and `sb_user` · create `ansible/inventory/host_vars/ipa-cloonar.sops.yaml` from the shipped `.example` with `sb_password`, `borg_passphrase`, `ipa_dm_password`, `ipa_admin_password`. Once ever: **confirm the snapshot plan on `u149513` is active** — `--append-only` is advisory when the sub-account has SFTP, so snapshots are the real backup control. No existing `secrets.yaml` was touched.

dominik.polakovics added 2 commits

2026-08-10 00:03:48 +02:00

feat(ansible): provision ipa.cloonar.com (AlmaLinux 10 + FreeIPA) from nb 194f100152

Adds the ansible/ tree that takes a freshly netcup-installed AlmaLinux 10 box
to a working IdM server in one command, plus the fatal DNS/rDNS preflight that
refuses to touch anything when the records are wrong.

Roles: preflight (read-only gate), base (hostname, static IPv6, swap, firewall,
dnf-automatic, alloy), borg (Storage Box sub-account, append-only repo, nightly
ipa-backup), freeipa (ipaserver wrapper + healthcheck), verify (acceptance).

389/636 are restricted to the WireGuard range rather than the internet, and
dnf-automatic applies OS updates while ipa-*, 389-ds-*, pki-* and *tomcat* stay
gated. The playbook holds no account-level credential: Storage Box sub-account
and DNS record creation stay manual on purpose.

scripts/ansible-check runs --syntax-check and ansible-lint, wired into the
pre-commit hook, since sync-host only ever walks hosts/*/ and this tree would
otherwise get no gate at all.

chore(sops): add a creation rule for ansible per-server secrets 70808a560a

Encrypts ansible/inventory/host_vars/*.sops.yaml to the admin keys only, with
no host key: these files are read on the control node and never shipped to the
target.

dominik.polakovics commented

2026-08-10 00:16:27 +02:00

Author

Owner

[autoland] verdict: pass

This was generated by AI while landing a PR.

CONCERNS (non-blocking) — landable.

Signal relied on: diff review only. labctl pr checks 314 returns state: none (this repo gates at commit time, not in CI), and this environment has no nix, ansible, or ansible-lint, so scripts/ansible-check could not be re-run here. The --syntax-check + ansible-lint --profile production pass reported in the PR body is the author's, unreproduced.

Checked: Conventional Commits title; Closes #313 present and valid; scope matches the issue (the deploy.yml paths-ignore line and the pre-commit wiring are both directly motivated, the latter explicitly invited by the issue); no account-level credential anywhere under ansible/ (the only lego-credentials hit is README prose explaining why it is not used) — issue acceptance criterion satisfied; all four .sops.yaml anchors (bitwarden, dominik, dominik2, nb) resolve against definitions earlier in the file; the alloy env-file mechanism matches utils/modules/alloy exactly (alloy-env → LOKI_PASSWORD → sys.env); branch merges cleanly into main.

Verified mechanically: the reworked scripts/pre-commit routing, exercised against eight representative staged-file sets — ansible/** and scripts/ansible-check route to the ansible gate with hosts='', shared paths still fan out to all hosts, hosts/<x>/ still narrows, unrelated paths still exit 0, and a mixed ansible/ + hosts/ diff runs both. No regression to the existing gate.

CONCERNS:

Merging closes #313 while most of its acceptance criteria are undemonstrated. A full run against a fresh AlmaLinux 10 box, the second-run-is-a-no-op check, dnf-automatic provably skipping the IPA stack, borg-mount, and an end-to-end restore all need the VPS, which does not exist yet. The PR is candid about this. Consider a follow-up issue for first-provisioning acceptance rather than letting the merge close the loop.
roles/verify/tasks/main.yml:16 — verify_healthcheck.stdout | from_json assumes ipa-healthcheck --output-type json writes JSON to stdout when --output-file is omitted, which the PR lists as unverified. If it does not, this raises a Jinja error mid-play rather than the intended message. Loud, not silent, but the diagnosis would be misleading.
The box ends up holding the fleet-wide Loki basic-auth credential. Disclosed in the README with base_alloy_enabled: false as the off-switch, and ipa.cloonar.com is not a third-party box, so the per-customer invariant holds as written — flagged only so the decision is consciously taken.

Nits, not concerns: scripts/pre-commit swallows ansible-check's "collections missing" hint via >/dev/null 2>&1 (the failure message already points at running it directly); &dominik and &nb are the same age key, so the new rule lists three distinct recipients, not four — matches the issue's wording, harmless.

[autoland] verdict: pass > *This was generated by AI while landing a PR.* **CONCERNS (non-blocking)** — landable. **Signal relied on:** diff review only. `labctl pr checks 314` returns `state: none` (this repo gates at commit time, not in CI), and this environment has no `nix`, `ansible`, or `ansible-lint`, so `scripts/ansible-check` could not be re-run here. The `--syntax-check` + `ansible-lint --profile production` pass reported in the PR body is the author's, unreproduced. **Checked:** Conventional Commits title; `Closes #313` present and valid; scope matches the issue (the `deploy.yml` `paths-ignore` line and the `pre-commit` wiring are both directly motivated, the latter explicitly invited by the issue); no account-level credential anywhere under `ansible/` (the only `lego-credentials` hit is README prose explaining why it is *not* used) — issue acceptance criterion satisfied; all four `.sops.yaml` anchors (`bitwarden`, `dominik`, `dominik2`, `nb`) resolve against definitions earlier in the file; the alloy env-file mechanism matches `utils/modules/alloy` exactly (`alloy-env` → `LOKI_PASSWORD` → `sys.env`); branch merges cleanly into `main`. **Verified mechanically:** the reworked `scripts/pre-commit` routing, exercised against eight representative staged-file sets — `ansible/**` and `scripts/ansible-check` route to the ansible gate with `hosts=''`, shared paths still fan out to all hosts, `hosts/<x>/` still narrows, unrelated paths still exit 0, and a mixed `ansible/` + `hosts/` diff runs both. No regression to the existing gate. **CONCERNS:** 1. **Merging closes #313 while most of its acceptance criteria are undemonstrated.** A full run against a fresh AlmaLinux 10 box, the second-run-is-a-no-op check, `dnf-automatic` provably skipping the IPA stack, `borg-mount`, and an end-to-end restore all need the VPS, which does not exist yet. The PR is candid about this. Consider a follow-up issue for first-provisioning acceptance rather than letting the merge close the loop. 2. **`roles/verify/tasks/main.yml:16`** — `verify_healthcheck.stdout | from_json` assumes `ipa-healthcheck --output-type json` writes JSON to stdout when `--output-file` is omitted, which the PR lists as unverified. If it does not, this raises a Jinja error mid-play rather than the intended message. Loud, not silent, but the diagnosis would be misleading. 3. **The box ends up holding the fleet-wide Loki basic-auth credential.** Disclosed in the README with `base_alloy_enabled: false` as the off-switch, and `ipa.cloonar.com` is not a third-party box, so the per-customer invariant holds as written — flagged only so the decision is consciously taken. Nits, not concerns: `scripts/pre-commit` swallows `ansible-check`'s "collections missing" hint via `>/dev/null 2>&1` (the failure message already points at running it directly); `&dominik` and `&nb` are the same age key, so the new rule lists three distinct recipients, not four — matches the issue's wording, harmless.

dominik.polakovics merged commit 228da4cf46 into main

2026-08-10 00:23:03 +02:00

dominik.polakovics referenced this pull request from a commit

2026-08-10 00:23:05 +02:00

Merge pull request 'feat(ansible): provision ipa.cloonar.com (AlmaLinux 10 + FreeIPA) from nb, with fatal DNS/rDNS preflight' (#314) from afk/313 into main

dominik.polakovics referenced this pull request

2026-08-10 00:23:26 +02:00

First provisioning run of ipa.cloonar.com — close out #313's acceptance criteria #315