Cloonar/nixos

Fork 0

feat(ansible): provision ipa.cloonar.com (AlmaLinux 10 + FreeIPA) from nb, with fatal DNS/rDNS preflight #313

New issue

Closed

opened 2026-08-09 23:23:57 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-08-09 23:23:57 +02:00

Owner

Build the Ansible tree that stands up a FreeIPA server on a netcup VPS from one command on nb, and the preflight that refuses to run when DNS isn't right.

The design below is settled. Where a decision looks arbitrary it usually isn't — the reasoning is given inline so it doesn't get "improved" back into the thing it replaced.

Context

FreeIPA is installed directly on the VPS, on AlmaLinux 10. Not on NixOS (nixpkgs has the client only — security.ipa is an enrolment module, the server is --disable-server), and not in a container (the image couples OS security updates to FreeIPA version bumps, and a NixOS host underneath would need ~8 GB because Nix evaluation spikes ~1 GB every five minutes on top of FreeIPA's 3–4 GB).

The target box is already bought: netcup, 4 GB, x86_64.

No custom image. An earlier plan built a qcow2 from this repo, because netcup provides no cloud-init datasource for custom uploads. That whole workstream is unnecessary: netcup ships AlmaLinux 10 as a stock image, and stock images are exactly the ones where SCP's hostname and SSH-key injection works. Install from the panel, then run the playbook.

Distribution is AlmaLinux 10, not Rocky. Upstream FreeIPA's freeipa-container repo carries Dockerfile.almalinux-{8,9,10}, Dockerfile.rocky-{8,9} — there is no rocky-10. Alma 10 is in upstream's test matrix; Rocky 10 is not. Rocky is also where the one IdM upgrade breakage we know of landed (9.5→9.6 broke ipa-server-upgrade, RHBZ #2350322). Pin 10.1 or newer: RHEL 10.0 shipped IdM only partially, with ipa-server-dns uninstallable. Not 9, because RHEL 9→10 in-place upgrade is unsupported for IdM (leapp excludes IdM servers), so the major chosen now is the major until a new server is built and migrated.

The install parameters, which are permanent

--hostname ipa.cloonar.com  --domain cloonar.com  --realm CLOONAR.COM

No --setup-dns. Two reasons: named costs memory that a 4 GB box does not have spare, and FreeIPA would want to publish _kerberos._udp.cloonar.com into a zone it does not own. The records aren't needed at all — see below.

FreeIPA cannot be renamed after ipa-server-install (RH solution 174733). The hostname/domain/realm triple is the one thing in this whole design that no revert reaches.

Why running alongside mail's KDC is safe

mail already serves realm CLOONAR.COM. Two KDCs for one realm is normally a race — clients take whichever answers. It is not a race here, because every krb5 client in the fleet has discovery disabled and the KDC pinned by hostname:

hosts/mail/modules/kerberos.nix:34            dns_lookup_kdc = false;  kdc = mail.cloonar.com
hosts/nb/modules/ldap-login.nix:91            dns_lookup_kdc = false;  kdc = mail.cloonar.com
hosts/nb/modules/ldap-login.nix:49            krb5_server = mail.cloonar.com    (SSSD)
hosts/web-arm/…/keycloak/default.nix:225      dns_lookup_kdc = false;  kdc = mail.cloonar.com

and kerberos.nix states outright that no _kerberos TXT or _kerberos._udp SRV records exist for cloonar.com.

So nothing can find the new KDC by accident. The two run in parallel for as long as wanted, and the eventual cutover is three lines of Nix delivered by bento, revertible with git revert. Cutover is out of scope for this issue — it needs an ADR first, because ADR-0023 explicitly rejected FreeIPA and has a live workstream (#299, #302, #308).

LDAP suffix coexistence is likewise fine: FreeIPA's 389DS serves dc=cloonar,dc=com on ipa.cloonar.com while mail's OpenLDAP serves the same suffix on ldap.cloonar.com. Different hosts, explicitly targeted clients.

Deliverable

ansible/
  ansible.cfg
  requirements.yml                          collections, pinned
  inventory/
    hosts.yml
    host_vars/ipa-cloonar.yml               public per-server config
    host_vars/ipa-cloonar.sops.yaml         per-server secrets
  playbooks/provision.yml
  roles/{preflight,base,borg,freeipa,verify}/
  README.md

One command from nb:

ansible-playbook -i ansible/inventory ansible/playbooks/provision.yml -l ipa-cloonar

Collections (all official, all pinned in requirements.yml):

Purpose	Collection
Secrets from this repo	`community.sops` (vars plugin, age)
FreeIPA install	`freeipa.ansible_freeipa` (`ipaserver` role)

No hetzner.hcloud. Sub-account creation is deliberately manual — see the security invariant below.

Config file shape

# ansible/inventory/host_vars/ipa-cloonar.yml
ipa_fqdn:         ipa.cloonar.com
ipa_domain:       cloonar.com
ipa_realm:        CLOONAR.COM
ipa_setup_dns:    false
net_iface:        ens3
net_ipv6:         "<addr>/64"
net_ipv6_gw:      "fe80::1"
sb_host:          u149513.your-backup.de
sb_user:          u149513-subNN
borg_repo:        "ssh://{{ sb_user }}@{{ sb_host }}:23/./ipa-cloonar/borg"
borg_remote_path: borg-1.4

# ansible/inventory/host_vars/ipa-cloonar.sops.yaml   (age; admin keys only)
sb_password:         …
borg_passphrase:     …
ipa_dm_password:     …
ipa_admin_password:  …

Add a .sops.yaml creation rule for ansible/inventory/host_vars/.*\.sops\.yaml$ with &bitwarden &dominik &dominik2 &nb — admin keys only, no host key. Do not modify any existing secrets.yaml; list what the operator must add and let them run sops themselves.

Security invariant — do not weaken this

No per-customer secrets file may contain a credential that reaches beyond that customer.

Every secret above is scoped to one server: one sub-account, one repo, one IdM instance. A leaked host_vars/*.sops.yaml compromises that customer and stops there.

This is why sub-account creation and DNS record creation are manual. Both are automatable — hetzner.hcloud.storage_box_subaccount exists and works, and the Hetzner DNS token is already in the repo as lego-credentials — but a Cloud API token is project-scoped, not resource-scoped, and a DNS token can mint a certificate for any name in the zone via DNS-01. Automating either would put a fleet-wide credential into a per-customer file.

The playbook therefore holds no account-level credential at all. The temptation to "just automate the sub-account too" will come back; this paragraph is the answer.

Roles

`preflight` — must run first and must be fatal

Explicitly requested. Refuse to touch anything unless all of these hold:

ipa_fqdn resolves to the target's IPv4 and IPv6
it is an A/AAAA record, not a CNAME — ipa-server-install rejects a CNAME hostname
reverse DNS on both v4 and v6 resolves back to ipa_fqdn
forward and reverse agree
the box is AlmaLinux ≥ 10.1
borg on the target is 1.4.x, and borg_remote_path matches
swap exists

Fail with a message naming the exact record that is wrong. rDNS is set in netcup's SCP and has no API, so a human has to fix it — the check exists to catch that before ipa-server-install fails halfway.

`base`

Hostname · static IPv6 (netcup is static only — no SLAAC, no DHCPv6, gateway fe80::1%<iface>; IPv4 comes from netcup's DHCP) · swapfile · packages including FUSE (python3-pyfuse3/fuse3, or borg mount is missing on the one box where restores happen) · firewall, with 389/636 restricted to WireGuard rather than the internet.

dnf-automatic in apply mode with exclude=ipa-* 389-ds-* pki-* *tomcat* in dnf.conf. This is the highest-value line in the build: unattended CVE patching for the OS, while the one component with no rollback path stays gated. ipa-server-upgrade runs in RPM %posttrans, is documented as "not intended to be executed by end-users", enters maintenance mode, and dnf history undo does not unwind 389DS schema migrations.

`borg`

Generate the client key on the target, then install it on the sub-account:

cat key.pub | ssh -p23 <subaccount>@<box> install-ssh-key

Hetzner confirms "This also works in subaccounts." It needs one password-authenticated session — the password is in sops, so use sshpass or ansible.builtin.expect; this stays automated.

Three traps:

Key format differs by port. Port 23 (borg) takes the ordinary one-line OpenSSH .pub; port 22 needs RFC4716 (ssh-keygen -e). Getting it wrong presents as an auth failure, not a format error.
Enabling SSH is not instant. Hetzner: "After you activate SSH, it can take a few minutes before you can access your Storage Box via TCP." Needs wait_for/retry, not an assumption.
Always pass --remote-path=borg-1.4. Hetzner installs borg 1.1, 1.2 (default) and 1.4, chosen per invocation. Without the pin you get 1.2 by accident and a future client bump breaks it silently.

Then borg init --append-only, a backup timer, and client-side helpers mirroring utils/modules/borgbackup.nix (borg-mount, borg-list, borg-restore) against the pushed passphrase file rather than sops-nix.

Append-only is advisory here. The sub-account has SFTP, so a compromised host can rewrite the repo's config and clear the flag. Snapshots are the real control — they are per-Storage-Box, taken by the main account, and invisible to sub-accounts. Setting the snapshot plan is a one-time console step; put "confirm the snapshot plan is active" in the README, because the whole backup threat model rests on it.

Record the installed NVRs of ipa-*, 389-ds-* and pki-* into the backup set: ipa-restore requires the same FQDN and the same package versions, so a restore is impossible without knowing them.

`freeipa`

Wrap freeipa.ansible_freeipa's ipaserver role with the parameters above.

ipa-server-install is not idempotent in the way Ansible implies. The role skips when IPA is already installed, but a half-failed install needs ipa-server-install --uninstall before a retry. Guard it explicitly and document the recovery in the README. Re-running the playbook will not fix a partial install.

Enable ipa-healthcheck.timer and ship its JSON to the existing alloy/Grafana stack.

`verify`

ipa-healthcheck clean · kinit smoke test · borg list against the real repo · borg-mount works.

Constraints

ansible/ must not go under hosts/. scripts/sync-host iterates $HOSTS_DIR/*/ and scripts/pre-commit runs ls hosts; a non-NixOS entry breaks both. Verified safe at the chosen path: pre-commit's shared regex is anchored (^\.sops\.yaml$, not a suffix match), so ansible/** matches neither it nor ^hosts/ and the hook exits 0. Consider adding an ansible-playbook --syntax-check / ansible-lint step to pre-commit so the tree gets some gate, since it currently gets none.
Every role idempotent and re-runnable; re-running is also the update path.
Do not modify secrets.yaml files. List required secrets and their location.
Comments sparse per CLAUDE.md — reasoning goes in the PR description, not the roles.

Manual steps to document in the README

Per server: install AlmaLinux 10 from SCP with the SSH key selected · set rDNS (v4 and v6) · create the Storage Box sub-account, setting the password yourself · create the A/AAAA records · write the two config files.

Once ever: confirm the snapshot plan on u149513.

Acceptance

ansible-playbook … -l ipa-cloonar takes a freshly installed AlmaLinux 10 box to a working IdM server.
Running it twice changes nothing the second time.
Preflight fails loudly, before any change, on: a CNAME hostname, missing rDNS, and forward/reverse mismatch — each tested deliberately.
The playbook holds no account-level credential; grep for an hcloud or DNS token in ansible/ returns nothing.
dnf-automatic applies an OS security update unattended and provably leaves ipa-*, 389-ds-*, pki-*, *tomcat* untouched.
Every borg invocation passes --remote-path=borg-1.4; borg-mount works on the target.
A restore has been performed end to end from a real archive.
mail's KDC is untouched and kinit against it still works — this issue changes nothing on the existing fleet.

Out of scope

Cutover from mail's KDC (needs an ADR — ADR-0023 rejected FreeIPA and its workstream is live in #299/#302/#308) · migrating the identity tenant out of OpenLDAP · enrolling nb/web-arm as IPA clients via security.ipa · replacing FreeIPA's Dogtag-issued web certificate with ACME.

Build the Ansible tree that stands up a FreeIPA server on a netcup VPS from one command on `nb`, and the preflight that refuses to run when DNS isn't right. The design below is settled. Where a decision looks arbitrary it usually isn't — the reasoning is given inline so it doesn't get "improved" back into the thing it replaced. ## Context FreeIPA is installed **directly on the VPS**, on AlmaLinux 10. Not on NixOS (nixpkgs has the client only — `security.ipa` is an enrolment module, the server is `--disable-server`), and not in a container (the image couples OS security updates to FreeIPA version bumps, and a NixOS host underneath would need ~8 GB because Nix evaluation spikes ~1 GB every five minutes on top of FreeIPA's 3–4 GB). The target box is already bought: netcup, 4 GB, x86_64. **No custom image.** An earlier plan built a qcow2 from this repo, because netcup provides no cloud-init datasource for custom uploads. That whole workstream is unnecessary: netcup ships **AlmaLinux 10 as a stock image**, and stock images are exactly the ones where SCP's hostname and SSH-key injection works. Install from the panel, then run the playbook. **Distribution is AlmaLinux 10, not Rocky.** Upstream FreeIPA's `freeipa-container` repo carries `Dockerfile.almalinux-{8,9,10}`, `Dockerfile.rocky-{8,9}` — **there is no `rocky-10`**. Alma 10 is in upstream's test matrix; Rocky 10 is not. Rocky is also where the one IdM upgrade breakage we know of landed (9.5→9.6 broke `ipa-server-upgrade`, RHBZ #2350322). Pin **10.1 or newer**: RHEL 10.0 shipped IdM only partially, with `ipa-server-dns` uninstallable. Not 9, because RHEL 9→10 in-place upgrade is unsupported for IdM (leapp excludes IdM servers), so the major chosen now is the major until a new server is built and migrated. ## The install parameters, which are permanent ``` --hostname ipa.cloonar.com --domain cloonar.com --realm CLOONAR.COM ``` **No `--setup-dns`.** Two reasons: `named` costs memory that a 4 GB box does not have spare, and FreeIPA would want to publish `_kerberos._udp.cloonar.com` into a zone it does not own. The records aren't needed at all — see below. FreeIPA **cannot be renamed** after `ipa-server-install` (RH solution 174733). The hostname/domain/realm triple is the one thing in this whole design that no revert reaches. ### Why running alongside mail's KDC is safe `mail` already serves realm `CLOONAR.COM`. Two KDCs for one realm is normally a race — clients take whichever answers. It is not a race here, because **every krb5 client in the fleet has discovery disabled and the KDC pinned by hostname**: ``` hosts/mail/modules/kerberos.nix:34 dns_lookup_kdc = false; kdc = mail.cloonar.com hosts/nb/modules/ldap-login.nix:91 dns_lookup_kdc = false; kdc = mail.cloonar.com hosts/nb/modules/ldap-login.nix:49 krb5_server = mail.cloonar.com (SSSD) hosts/web-arm/…/keycloak/default.nix:225 dns_lookup_kdc = false; kdc = mail.cloonar.com ``` and `kerberos.nix` states outright that no `_kerberos` TXT or `_kerberos._udp` SRV records exist for `cloonar.com`. So nothing can find the new KDC by accident. The two run in parallel for as long as wanted, and the eventual cutover is three lines of Nix delivered by bento, revertible with `git revert`. **Cutover is out of scope for this issue** — it needs an ADR first, because ADR-0023 explicitly rejected FreeIPA and has a live workstream (#299, #302, #308). LDAP suffix coexistence is likewise fine: FreeIPA's 389DS serves `dc=cloonar,dc=com` on `ipa.cloonar.com` while `mail`'s OpenLDAP serves the same suffix on `ldap.cloonar.com`. Different hosts, explicitly targeted clients. ## Deliverable ``` ansible/ ansible.cfg requirements.yml collections, pinned inventory/ hosts.yml host_vars/ipa-cloonar.yml public per-server config host_vars/ipa-cloonar.sops.yaml per-server secrets playbooks/provision.yml roles/{preflight,base,borg,freeipa,verify}/ README.md ``` One command from `nb`: ``` ansible-playbook -i ansible/inventory ansible/playbooks/provision.yml -l ipa-cloonar ``` Collections (all official, all pinned in `requirements.yml`): | Purpose | Collection | |---|---| | Secrets from this repo | `community.sops` (vars plugin, age) | | FreeIPA install | `freeipa.ansible_freeipa` (`ipaserver` role) | **No `hetzner.hcloud`.** Sub-account creation is deliberately manual — see the security invariant below. ### Config file shape ```yaml # ansible/inventory/host_vars/ipa-cloonar.yml ipa_fqdn: ipa.cloonar.com ipa_domain: cloonar.com ipa_realm: CLOONAR.COM ipa_setup_dns: false net_iface: ens3 net_ipv6: "<addr>/64" net_ipv6_gw: "fe80::1" sb_host: u149513.your-backup.de sb_user: u149513-subNN borg_repo: "ssh://{{ sb_user }}@{{ sb_host }}:23/./ipa-cloonar/borg" borg_remote_path: borg-1.4 ``` ```yaml # ansible/inventory/host_vars/ipa-cloonar.sops.yaml (age; admin keys only) sb_password: … borg_passphrase: … ipa_dm_password: … ipa_admin_password: … ``` Add a `.sops.yaml` creation rule for `ansible/inventory/host_vars/.*\.sops\.yaml$` with `&bitwarden &dominik &dominik2 &nb` — **admin keys only, no host key.** Do not modify any existing `secrets.yaml`; list what the operator must add and let them run `sops` themselves. ## Security invariant — do not weaken this **No per-customer secrets file may contain a credential that reaches beyond that customer.** Every secret above is scoped to one server: one sub-account, one repo, one IdM instance. A leaked `host_vars/*.sops.yaml` compromises that customer and stops there. This is why sub-account creation and DNS record creation are **manual**. Both are automatable — `hetzner.hcloud.storage_box_subaccount` exists and works, and the Hetzner DNS token is already in the repo as `lego-credentials` — but a Cloud API token is project-scoped, not resource-scoped, and a DNS token can mint a certificate for any name in the zone via DNS-01. Automating either would put a fleet-wide credential into a per-customer file. The playbook therefore holds **no account-level credential at all**. The temptation to "just automate the sub-account too" will come back; this paragraph is the answer. ## Roles ### `preflight` — must run first and must be fatal Explicitly requested. Refuse to touch anything unless all of these hold: - `ipa_fqdn` resolves to the target's IPv4 **and** IPv6 - it is an **A/AAAA record, not a CNAME** — `ipa-server-install` rejects a CNAME hostname - **reverse DNS** on both v4 and v6 resolves back to `ipa_fqdn` - forward and reverse agree - the box is AlmaLinux ≥ 10.1 - borg on the target is 1.4.x, and `borg_remote_path` matches - swap exists Fail with a message naming the exact record that is wrong. rDNS is set in netcup's SCP and has no API, so a human has to fix it — the check exists to catch that before `ipa-server-install` fails halfway. ### `base` Hostname · static IPv6 (netcup is **static only** — no SLAAC, no DHCPv6, gateway `fe80::1%<iface>`; IPv4 comes from netcup's DHCP) · swapfile · packages including **FUSE** (`python3-pyfuse3`/`fuse3`, or `borg mount` is missing on the one box where restores happen) · firewall, with **389/636 restricted to WireGuard** rather than the internet. `dnf-automatic` in **apply** mode with `exclude=ipa-* 389-ds-* pki-* *tomcat*` in `dnf.conf`. This is the highest-value line in the build: unattended CVE patching for the OS, while the one component with no rollback path stays gated. `ipa-server-upgrade` runs in RPM `%posttrans`, is documented as "not intended to be executed by end-users", enters maintenance mode, and `dnf history undo` does **not** unwind 389DS schema migrations. ### `borg` Generate the client key on the target, then install it on the sub-account: ``` cat key.pub | ssh -p23 <subaccount>@<box> install-ssh-key ``` Hetzner confirms *"This also works in subaccounts."* It needs one password-authenticated session — the password is in sops, so use `sshpass` or `ansible.builtin.expect`; this stays automated. Three traps: - **Key format differs by port.** Port 23 (borg) takes the ordinary one-line OpenSSH `.pub`; port 22 needs RFC4716 (`ssh-keygen -e`). Getting it wrong presents as an auth failure, not a format error. - **Enabling SSH is not instant.** Hetzner: *"After you activate SSH, it can take a few minutes before you can access your Storage Box via TCP."* Needs `wait_for`/retry, not an assumption. - **Always pass `--remote-path=borg-1.4`.** Hetzner installs borg 1.1, 1.2 (default) and 1.4, chosen per invocation. Without the pin you get 1.2 by accident and a future client bump breaks it silently. Then `borg init --append-only`, a backup timer, and client-side helpers mirroring `utils/modules/borgbackup.nix` (`borg-mount`, `borg-list`, `borg-restore`) against the pushed passphrase file rather than sops-nix. **Append-only is advisory here.** The sub-account has SFTP, so a compromised host can rewrite the repo's `config` and clear the flag. Snapshots are the real control — they are per-Storage-Box, taken by the main account, and **invisible to sub-accounts**. Setting the snapshot plan is a one-time console step; put "confirm the snapshot plan is active" in the README, because the whole backup threat model rests on it. Record the installed NVRs of `ipa-*`, `389-ds-*` and `pki-*` into the backup set: `ipa-restore` requires the **same FQDN and the same package versions**, so a restore is impossible without knowing them. ### `freeipa` Wrap `freeipa.ansible_freeipa`'s `ipaserver` role with the parameters above. **`ipa-server-install` is not idempotent in the way Ansible implies.** The role skips when IPA is already installed, but a *half-failed* install needs `ipa-server-install --uninstall` before a retry. Guard it explicitly and document the recovery in the README. Re-running the playbook will not fix a partial install. Enable `ipa-healthcheck.timer` and ship its JSON to the existing alloy/Grafana stack. ### `verify` `ipa-healthcheck` clean · `kinit` smoke test · `borg list` against the real repo · `borg-mount` works. ## Constraints - **`ansible/` must not go under `hosts/`.** `scripts/sync-host` iterates `$HOSTS_DIR/*/` and `scripts/pre-commit` runs `ls hosts`; a non-NixOS entry breaks both. Verified safe at the chosen path: `pre-commit`'s shared regex is anchored (`^\.sops\.yaml$`, not a suffix match), so `ansible/**` matches neither it nor `^hosts/` and the hook exits 0. Consider adding an `ansible-playbook --syntax-check` / `ansible-lint` step to `pre-commit` so the tree gets *some* gate, since it currently gets none. - Every role idempotent and re-runnable; re-running is also the update path. - Do not modify `secrets.yaml` files. List required secrets and their location. - Comments sparse per CLAUDE.md — reasoning goes in the PR description, not the roles. ## Manual steps to document in the README **Per server:** install AlmaLinux 10 from SCP with the SSH key selected · set rDNS (v4 and v6) · create the Storage Box sub-account, setting the password yourself · create the A/AAAA records · write the two config files. **Once ever:** confirm the snapshot plan on `u149513`. ## Acceptance - `ansible-playbook … -l ipa-cloonar` takes a freshly installed AlmaLinux 10 box to a working IdM server. - Running it twice changes nothing the second time. - Preflight fails loudly, before any change, on: a CNAME hostname, missing rDNS, and forward/reverse mismatch — each tested deliberately. - The playbook holds no account-level credential; `grep` for an hcloud or DNS token in `ansible/` returns nothing. - `dnf-automatic` applies an OS security update unattended and provably leaves `ipa-*`, `389-ds-*`, `pki-*`, `*tomcat*` untouched. - Every borg invocation passes `--remote-path=borg-1.4`; `borg-mount` works on the target. - A restore has been performed end to end from a real archive. - `mail`'s KDC is untouched and `kinit` against it still works — this issue changes nothing on the existing fleet. ## Out of scope Cutover from `mail`'s KDC (needs an ADR — ADR-0023 rejected FreeIPA and its workstream is live in #299/#302/#308) · migrating the identity tenant out of OpenLDAP · enrolling `nb`/`web-arm` as IPA clients via `security.ipa` · replacing FreeIPA's Dogtag-issued web certificate with ACME.

dominik.polakovics added the

ready-for-agent

enhancement

labels

2026-08-09 23:23:57 +02:00

dominik.polakovics referenced this issue from a pull request that will close it,

2026-08-10 00:03:48 +02:00

feat(ansible): provision ipa.cloonar.com (AlmaLinux 10 + FreeIPA) from nb, with fatal DNS/rDNS preflight #314

dominik.polakovics referenced this issue from a pull request that will close it,

2026-08-10 00:16:27 +02:00

feat(ansible): provision ipa.cloonar.com (AlmaLinux 10 + FreeIPA) from nb, with fatal DNS/rDNS preflight #314

dominik.polakovics closed this issue