Cloonar/nixos
3
0
Fork 0
Code Issues 31 Pull requests 7 Projects Releases Packages Wiki Activity Actions

Three verification spikes before the Kerberos-authoritative build (M1) #301

New issue
Closed
opened 2026-08-08 13:29:58 +02:00 by dominik.polakovics · 1 comment
dominik.polakovics commented 2026-08-08 13:29:58 +02:00
Owner
Copy link

M1 of #299. Three cheap questions whose answers change what gets built. All human-run; none needs a PR.

1. Does Dovecot pass PAM the bare local part or the full address?

Decides whether the Kerberos passdb is driver = pam with pam_krb5 (simple, gets keytab verification free) or a checkpassword script (~30 lines, full control of the mapping).

auth_username_format is unset in hosts/mail/modules/, so the 2.3 default %Lu applies — full lowercased user@domain — and it must stay that way because mail_location uses %d/%n and every LDAP filter keys on mail=%u. If PAM receives the full address, pam_krb5 forms dominik.polakovics@cloonar.com@CLOONAR.COM and (b) is dead. Giving Dovecot its own principal name is not an escape: one human must have exactly one principal, or the drift is back.

Safe test — put the PAM passdb last, so successful logins stop at LDAP and never reach it, and pam_deny means it cannot authenticate anyone regardless. mail rebuilds from /var/bento every five minutes (utils/bento.nix:6,10), so stop bento-upgrade.timer first and let it restore the config afterwards.

security.pam.services.dovecot-pamtest.text = ''
  auth     required pam_exec.so ${pkgs.writeShellScript "pamtest-log" ''
    ${pkgs.util-linux}/bin/logger -t dovecot-pamtest "PAM_USER=[$PAM_USER]"
  ''}
  auth     required pam_deny.so
  account  required pam_permit.so
'';

passdb {
  driver = pam
  args = dovecot-pamtest
}

Then doveadm auth test dominik.polakovics@cloonar.com definitely-wrong and read journalctl -t dovecot-pamtest. LDAP rejects the wrong password, result_failure defaults to continue, and the chain falls through to PAM.

Also worth running once with a correct password for an account on another domain: it should succeed at the LDAP passdb and log nothing, confirming the chain order before anything real is built on it.

2. Can SDDM render an expired-password prompt?

The admin-reset flow is kadmin cpw with the password marked expired, so the user is forced to set their own at next login. That is a PAM CHANGE_EXPIRED_AUTHTOK conversation delivered by the greeter. GDM handles it; SDDM has a patchy history.

kadmin.local -q "cpw -e now dominik.polakovics", then log in at the greeter. If it prompts, the flow works. If it just says login failed, the user has no way forward and the design needs a different greeter or unexpired temporary passwords.

3. Does useKerberosForPasswordAuthentication fall back to an LDAP bind?

It should not — the documented behaviour is that it replaces LDAP password authentication. Confirm it, because the sequencing depends on it: if there is no fallback, then flipping it locks out every federated user who has no principal yet, and M4 cannot precede M6.

Test with a user that has an LDAP password and no principal. Being certain matters more than the answer.

M1 of #299. Three cheap questions whose answers change what gets built. All human-run; none needs a PR. ## 1. Does Dovecot pass PAM the bare local part or the full address? Decides whether the Kerberos passdb is `driver = pam` with pam_krb5 (simple, gets keytab verification free) or a `checkpassword` script (~30 lines, full control of the mapping). `auth_username_format` is unset in `hosts/mail/modules/`, so the 2.3 default `%Lu` applies — full lowercased `user@domain` — and it must stay that way because `mail_location` uses `%d`/`%n` and every LDAP filter keys on `mail=%u`. If PAM receives the full address, pam_krb5 forms `dominik.polakovics@cloonar.com@CLOONAR.COM` and (b) is dead. Giving Dovecot its own principal name is not an escape: one human must have exactly one principal, or the drift is back. Safe test — put the PAM passdb **last**, so successful logins stop at LDAP and never reach it, and `pam_deny` means it cannot authenticate anyone regardless. `mail` rebuilds from `/var/bento` every five minutes (`utils/bento.nix:6,10`), so stop `bento-upgrade.timer` first and let it restore the config afterwards. ``` security.pam.services.dovecot-pamtest.text = '' auth required pam_exec.so ${pkgs.writeShellScript "pamtest-log" '' ${pkgs.util-linux}/bin/logger -t dovecot-pamtest "PAM_USER=[$PAM_USER]" ''} auth required pam_deny.so account required pam_permit.so ''; ``` ``` passdb { driver = pam args = dovecot-pamtest } ``` Then `doveadm auth test dominik.polakovics@cloonar.com definitely-wrong` and read `journalctl -t dovecot-pamtest`. LDAP rejects the wrong password, `result_failure` defaults to `continue`, and the chain falls through to PAM. Also worth running once with a **correct** password for an account on another domain: it should succeed at the LDAP passdb and log nothing, confirming the chain order before anything real is built on it. ## 2. Can SDDM render an expired-password prompt? The admin-reset flow is `kadmin cpw` with the password marked expired, so the user is forced to set their own at next login. That is a PAM `CHANGE_EXPIRED_AUTHTOK` conversation delivered by the greeter. GDM handles it; SDDM has a patchy history. `kadmin.local -q "cpw -e now dominik.polakovics"`, then log in at the greeter. If it prompts, the flow works. If it just says login failed, the user has no way forward and the design needs a different greeter or unexpired temporary passwords. ## 3. Does `useKerberosForPasswordAuthentication` fall back to an LDAP bind? It should not — the documented behaviour is that it replaces LDAP password authentication. Confirm it, because the sequencing depends on it: if there is no fallback, then flipping it locks out every federated user who has no principal yet, and M4 cannot precede M6. Test with a user that has an LDAP password and no principal. Being certain matters more than the answer.
dominik.polakovics commented 2026-08-09 01:45:31 +02:00
Author
Owner
Copy link

This was generated by AI during triage.

All three spikes run — results

Run 2026-08-08. Two of the three came back differently from what the issue predicted, and both change the build.

Spike 1 — Dovecot hands PAM the full address

Measured on a throwaway Dovecot instance under its own base_dir; production Dovecot was never reconfigured, restarted, or contacted.

PAM_USER=[dominik.polakovics@cloonar.com]
PAM_USER=[someone@superbros.tv]

Confirmed independently by auth_debug: pam(dominik.polakovics@cloonar.com): lookup service=dovecot-pamtest. Host runs Dovecot 2.3.21.1 (pkgs.dovecot_2_3, pinned by stateVersion 22.11 < 26.05). The second-domain result shows the format is not tenant-dependent.

→ M3 is a checkpassword script. The pam_krb5 route is dead. It would form dominik.polakovics@cloonar.com@CLOONAR.COM, which cannot resolve. There is no escape: auth_username_format must stay %Lu (mail_location uses %d/%n, filters key on mail=%u, and the primary passdb's base = ou=users,dc=%Dd), and 2.3's PAM passdb has no per-passdb username rewriting — its documented args are session, setcred, max_requests, failure_show_msg, cache_key, %s, override_fields. That only became per-passdb in 2.4.

Two things for whoever builds M3:

  • The script must return user unknown, not auth failed, for non-identity-tenant addresses, so result_failure = continue carries them to the LDAP passdbs untouched.
  • #301 listed "gets keytab verification free" as a pam_krb5 benefit. Losing it is not free — a bare kinit check trusts whatever answers as the KDC. The script must verify against a keytab or the exposure must be accepted deliberately. On mail the KDC is local so the real risk is near nil, but ADR-0023 sells this as a reference architecture and a split deployment inherits it.

Spike 2 — SDDM renders the prompt; two bugs found and fixed

The greeter question is answered: yes. SDDM renders CHANGE_EXPIRED_AUTHTOK correctly. The "different greeter" contingency is dead, and so is the theme suspicion — the minimal where-is-my-sddm-theme handles it.

Getting there required fixing two real defects:

#305 — kpasswd (464) was firewalled. kadmind listened on 464 and 749, but nixos-fw allowed only 88. SSSD's chpass_provider = krb5 with no krb5_kpasswd set assumes the service runs on the KDC, so every change aimed at a dropped port. The failure was silent: the greeter rendered the prompt, accepted a new password, reported nothing, and logged the user in from SSSD's offline cache (krb5_store_password_if_offline = true, ldap-login.nix:52) — while the principal kept REQUIRES_PWCHANGE and its old key.

Under ADR-0023, where Kerberos is the only credential, that is a user who believes they rotated a compromised password and did not. M5's Required Action must handle a failed kpasswd explicitly rather than trusting a clean return.

#306 — the password stack prompted twice. Old password ×2, new ×4. pam_chauthtok() runs the stack in two passes (PAM_PRELIM_CHECK, PAM_UPDATE_AUTHTOK) and both pam_unix (10200) and pam_sss (10900) prompted in each, because pam_sss carries no use_authtok. pam_unix can never succeed for these accounts — they are declared locally with no password on purpose (ldap-login.nix:120-122) so SDDM lists them. Fixed by ordering pam_sss first so it short-circuits as sufficient.

Not a correctness bug: kvno stepped 4 → 5, not 4 → 6, across a full doubled sequence.

Correction to this issue's setup text: kadmin.local -q "cpw -e now <princ>" does not work. -e on cpw is the keysalt list. Use cpw -pw <temp> then modprinc +needchange.

Spike 3 — no fallback

With useKerberosForPasswordAuthentication = true on the cloonar realm:

User Password Result
dominik.polakovics (has principal) Kerberos works
dominik.polakovics (has principal) LDAP rejected
office@cloonar.com (no principal) LDAP rejected

It replaces LDAP password authentication entirely. The documented behaviour holds.

Getting a clean answer needed two prerequisites nobody had identified — see below.

Two findings nobody was looking for

web-arm had never been able to reach the KDC. keycloak/default.nix:153-156 states the assumption outright — "Accepting a ticket needs no KDC contact, so only the realm mapping matters." True for SPNEGO, which validates against the keytab locally. realms."CLOONAR.COM".kdc was configured but never exercised. useKerberosForPasswordAuthentication is the first thing that uses it. M4 has an unstated prerequisite: a firewall rule permitting web-arm → mail:88, which lives in the Hetzner console, outside git. That comment is now misleading and should be corrected.

The dual-stack v6 trap from #97 bit a second consumer. mail.cloonar.com is dual-stack (ADR-0016) and web-arm has v6 egress (ADR-0010), so RFC 6724 makes clients prefer 2a01:4f8:c012:9d85::2. With only an IPv4 firewall rule, kinit worked (MIT krb5 iterates all addresses and falls back) while Keycloak hung — the JVM's Kerberos client has no v6→v4 fallback, exactly as FreeScout's IMAP client had none in #97. Resolved by opening the v6 source.

Worth noting the residual: the JVM will hang the same way any time the v6 path breaks, while kinit on the same host keeps working. ADR-0016's precedent for this class was to pin the consumer to IPv4. Also, the current rules are TCP-only on 88/464 — Kerberos tries UDP first and falls back to TCP for oversized responses, so this works until a response outgrows a datagram.

Consequence: the milestone order is wrong

Spike 3 does more than reorder M4 and M6 — it shows no ordering of them as written works:

  • M4 before M6 — Keycloak goes Kerberos-only while most users have no principal. Everyone un-enrolled loses gitea, grafana, freescout, immich, nextcloud and Matrix. Demonstrated with office@cloonar.com.
  • M6 before M4 — enrolment is atomic per ADR-0023 decision 4 and ends by locking userPassword. Keycloak is still binding to LDAP at that point, so each user loses web SSO the moment they are enrolled.

useKerberosForPasswordAuthentication is a realm-level flag and enrolment is explicitly per-user. A per-user cutover cannot be expressed through a fleet-wide switch.

Tracked as a design question in the follow-up issue rather than here. #299's milestone list has been corrected.

Smaller follow-ups

  • /etc/pam.d/passwd on nb likely has the same unix → sss ordering #306 fixed for login. Left alone deliberately: setting .order on a service whose sss rule does not exist would create a rule with no modulePath.
  • ldap-login.nix:21 sets config_file_version = 2, which SSSD 2.13 rejects — log noise only.
  • kerberos.nix:5 still describes the pre-#305 Hetzner boundary.
  • ADR-0023 is ambiguous in-tree: turn-runner.nix:19 and api.reptide.eu.nix:28 use the bare number for fit's ADR-0023, and this repo now has its own.
  • Ticket life deferred by maintainer decision. The principal carries Maximum ticket life: 1 day / Maximum renewable life: 0 against ADR-0023 decision 9's 10h / 7d. kdc.conf realm defaults do not apply retroactively, so enrolment must set these per principal. Recorded on #299 so it is not lost.

All three spikes are answered. Closing.

> *This was generated by AI during triage.* ## All three spikes run — results Run 2026-08-08. Two of the three came back differently from what the issue predicted, and both change the build. ### Spike 1 — Dovecot hands PAM the **full address** Measured on a throwaway Dovecot instance under its own `base_dir`; production Dovecot was never reconfigured, restarted, or contacted. ``` PAM_USER=[dominik.polakovics@cloonar.com] PAM_USER=[someone@superbros.tv] ``` Confirmed independently by `auth_debug`: `pam(dominik.polakovics@cloonar.com): lookup service=dovecot-pamtest`. Host runs Dovecot **2.3.21.1** (`pkgs.dovecot_2_3`, pinned by `stateVersion 22.11 < 26.05`). The second-domain result shows the format is not tenant-dependent. **→ M3 is a `checkpassword` script. The pam_krb5 route is dead.** It would form `dominik.polakovics@cloonar.com@CLOONAR.COM`, which cannot resolve. There is no escape: `auth_username_format` must stay `%Lu` (`mail_location` uses `%d`/`%n`, filters key on `mail=%u`, and the primary passdb's `base = ou=users,dc=%Dd`), and 2.3's PAM passdb has no per-passdb username rewriting — its documented args are `session`, `setcred`, `max_requests`, `failure_show_msg`, `cache_key`, `%s`, `override_fields`. That only became per-passdb in 2.4. Two things for whoever builds M3: - The script must return **user unknown**, not auth failed, for non-identity-tenant addresses, so `result_failure = continue` carries them to the LDAP passdbs untouched. - #301 listed "gets keytab verification free" as a pam_krb5 benefit. Losing it is not free — a bare `kinit` check trusts whatever answers as the KDC. The script must verify against a keytab or the exposure must be accepted deliberately. On `mail` the KDC is local so the real risk is near nil, but ADR-0023 sells this as a reference architecture and a split deployment inherits it. ### Spike 2 — SDDM renders the prompt; two bugs found and fixed **The greeter question is answered: yes.** SDDM renders `CHANGE_EXPIRED_AUTHTOK` correctly. The "different greeter" contingency is dead, and so is the theme suspicion — the minimal `where-is-my-sddm-theme` handles it. Getting there required fixing two real defects: **#305 — kpasswd (464) was firewalled.** `kadmind` listened on 464 and 749, but `nixos-fw` allowed only 88. SSSD's `chpass_provider = krb5` with no `krb5_kpasswd` set assumes the service runs on the KDC, so every change aimed at a dropped port. **The failure was silent**: the greeter rendered the prompt, accepted a new password, reported nothing, and logged the user in from SSSD's offline cache (`krb5_store_password_if_offline = true`, `ldap-login.nix:52`) — while the principal kept `REQUIRES_PWCHANGE` and its old key. Under ADR-0023, where Kerberos is the only credential, that is a user who believes they rotated a compromised password and did not. **M5's Required Action must handle a failed kpasswd explicitly rather than trusting a clean return.** **#306 — the password stack prompted twice.** Old password ×2, new ×4. `pam_chauthtok()` runs the stack in two passes (`PAM_PRELIM_CHECK`, `PAM_UPDATE_AUTHTOK`) and both `pam_unix` (10200) and `pam_sss` (10900) prompted in each, because `pam_sss` carries no `use_authtok`. `pam_unix` can never succeed for these accounts — they are declared locally with no password on purpose (`ldap-login.nix:120-122`) so SDDM lists them. Fixed by ordering `pam_sss` first so it short-circuits as `sufficient`. Not a correctness bug: kvno stepped **4 → 5**, not 4 → 6, across a full doubled sequence. **Correction to this issue's setup text:** `kadmin.local -q "cpw -e now <princ>"` does not work. `-e` on `cpw` is the keysalt list. Use `cpw -pw <temp>` then `modprinc +needchange`. ### Spike 3 — **no fallback** With `useKerberosForPasswordAuthentication = true` on the `cloonar` realm: | User | Password | Result | | --- | --- | --- | | `dominik.polakovics` (has principal) | Kerberos | **works** | | `dominik.polakovics` (has principal) | LDAP | rejected | | `office@cloonar.com` (no principal) | LDAP | rejected | **It replaces LDAP password authentication entirely.** The documented behaviour holds. Getting a clean answer needed two prerequisites nobody had identified — see below. ## Two findings nobody was looking for **`web-arm` had never been able to reach the KDC.** `keycloak/default.nix:153-156` states the assumption outright — *"Accepting a ticket needs no KDC contact, so only the realm mapping matters."* True for SPNEGO, which validates against the keytab locally. `realms."CLOONAR.COM".kdc` was configured but never exercised. `useKerberosForPasswordAuthentication` is the first thing that uses it. **M4 has an unstated prerequisite: a firewall rule permitting web-arm → mail:88, which lives in the Hetzner console, outside git.** That comment is now misleading and should be corrected. **The dual-stack v6 trap from #97 bit a second consumer.** `mail.cloonar.com` is dual-stack (ADR-0016) and `web-arm` has v6 egress (ADR-0010), so RFC 6724 makes clients prefer `2a01:4f8:c012:9d85::2`. With only an IPv4 firewall rule, `kinit` worked (MIT krb5 iterates all addresses and falls back) while **Keycloak hung** — the JVM's Kerberos client has no v6→v4 fallback, exactly as FreeScout's IMAP client had none in #97. Resolved by opening the v6 source. Worth noting the residual: the JVM will hang the same way any time the v6 path breaks, while `kinit` on the same host keeps working. ADR-0016's precedent for this class was to pin the consumer to IPv4. Also, the current rules are **TCP-only** on 88/464 — Kerberos tries UDP first and falls back to TCP for oversized responses, so this works until a response outgrows a datagram. ## Consequence: the milestone order is wrong Spike 3 does more than reorder M4 and M6 — it shows no ordering of them as written works: - **M4 before M6** — Keycloak goes Kerberos-only while most users have no principal. Everyone un-enrolled loses gitea, grafana, freescout, immich, nextcloud and Matrix. Demonstrated with `office@cloonar.com`. - **M6 before M4** — enrolment is atomic per ADR-0023 decision 4 and ends by locking `userPassword`. Keycloak is still binding to LDAP at that point, so each user loses web SSO *the moment they are enrolled*. `useKerberosForPasswordAuthentication` is a **realm-level flag** and enrolment is explicitly per-user. A per-user cutover cannot be expressed through a fleet-wide switch. Tracked as a design question in the follow-up issue rather than here. #299's milestone list has been corrected. ## Smaller follow-ups - `/etc/pam.d/passwd` on `nb` likely has the same `unix` → `sss` ordering #306 fixed for `login`. Left alone deliberately: setting `.order` on a service whose `sss` rule does not exist would create a rule with no `modulePath`. - `ldap-login.nix:21` sets `config_file_version = 2`, which SSSD 2.13 rejects — log noise only. - `kerberos.nix:5` still describes the pre-#305 Hetzner boundary. - `ADR-0023` is ambiguous in-tree: `turn-runner.nix:19` and `api.reptide.eu.nix:28` use the bare number for **fit's** ADR-0023, and this repo now has its own. - Ticket life deferred by maintainer decision. The principal carries `Maximum ticket life: 1 day` / `Maximum renewable life: 0` against ADR-0023 decision 9's `10h` / `7d`. `kdc.conf` realm defaults do **not** apply retroactively, so enrolment must set these per principal. Recorded on #299 so it is not lost. All three spikes are answered. Closing.
dominik.polakovics 2026-08-09 01:45:44 +02:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
fix/wow-config-self-heal
docs/web-arm-hetzner-egress-loss
afk/385
afk/360
afk/353
fix/onecli-session-jwt
fix/onecli-db-password-credential
lab/onecli-dashboard-hairpin
afk/319
lab/20260808-1423-pam-order
feat/typo3-sso-clients
lab/20260807-2348
lab/20260807-0936-sddm
feat/nb-transcribe-speakers
afk/243
lab/20260726-1358
feat/dev-new-system-dev-tools
afk/215
fix/fw-disk-reclaim
fix/lab-pin-opus-4-8
feat/diag-on-internal-vms
feat/fw-cpu-priorities-and-nix-shell
No results found.
Labels
Clear labels   
bug

   
enhancement

   
in-progress

   
needs-info

   
needs-triage

   
p0

   
ready-for-agent

   
ready-for-human

   
wontfix
No labels
bug
enhancement
in-progress
needs-info
needs-triage
p0
ready-for-agent
ready-for-human
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Cloonar/nixos#301
No description provided.