M3 (decision gate): Keycloak at sso.cloonar.com beside Authelia, declarative realm + SPNEGO #271

New issue

Closed

opened 2026-08-07 13:57:30 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-08-07 13:57:30 +02:00

Owner

Part of #266 (M3 of M1–M6). Follows #267 (KDC on mail, complete) and #269 (SSSD + Kerberos login on nb, complete — desktop login yields a TGT in KCM).

M3 is the decision gate for the whole workstream. Keycloak comes up at sso.cloonar.com beside Authelia, which keeps serving all of production untouched. The single question M3 answers is: does a nb desktop login get the user into Keycloak with no password prompt? If yes, M4–M6 proceed. If no, the workstream stops and nothing has been cut over. Design context: docs/adr/0022-keycloak-and-scoped-kerberos.md.

Nothing is cut over in this issue

hosts/web-arm/modules/authelia.nix is not touched. auth.cloonar.com keeps serving every existing client.
No OIDC clients are registered in Keycloak (that is M4).
No oauth2-proxy, no changes to any existing nginx vhost, nothing retired.
Two IdPs running side by side is the deliberate design, not a transitional accident.

Prerequisites already in place

keycloak-db-password, keycloak-bootstrap-admin, keycloak-keytab-b64 are committed in hosts/web-arm/secrets.yaml.
The keytab holds HTTP/sso.cloonar.com@CLOONAR.COM, kvno 2, enctypes aes256-cts-hmac-sha1-96 + aes128-cts-hmac-sha1-96.
LDAP read access: bind as cn=linuxbind,ou=system,ou=users,dc=cloonar,dc=com, whose password is already in hosts/web-arm/secrets.yaml as linuxbind-password. Federation is READ_ONLY, so no new system DN and no LDAP write is needed.
PostgreSQL already runs on web-arm (hosts/web-arm/modules/postgresql.nix).
Headroom is not a concern: web-arm measured 10 GB available of 15.5 GB.
DNS for sso.cloonar.com is a human step (below). It does not block the build: ACME here is DNS-01 via the Hetzner API (utils/modules/lego/lego.nix:11), so the certificate issues without an A record.

Scope

Create hosts/web-arm/modules/keycloak/ (module + realm JSON) and import it from hosts/web-arm/configuration.nix.

1. The Keycloak service

services.keycloak with the PostgreSQL backend and database.passwordFile = the keycloak-db-password secret.
initialAdminPassword must stay unset. The module renders it into Environment= in the unit file, which is world-readable in the Nix store. Supply KC_BOOTSTRAP_ADMIN_USERNAME and KC_BOOTSTRAP_ADMIN_PASSWORD instead via systemd.services.keycloak.serviceConfig.EnvironmentFile pointed at the keycloak-bootstrap-admin secret. Leaving the option null means the module emits no competing Environment= lines.
Keycloak 26 hostname semantics: settings.hostname is the full external URL. HTTP-only on loopback behind nginx, with proxy-headers = "xforwarded" so it trusts the reverse proxy's X-Forwarded-*.

2. The keytab

keycloak-keytab-b64 is base64. Decode it to a stable runtime path owned by the keycloak user, mode 0400, in an ExecStartPre. hosts/mail/modules/dovecot.nix:324 is the existing in-repo pattern for rendering a secret into a runtime file at service start. Do not write it anywhere in /nix/store. The realm JSON references this path, so it must be fixed and known at build time.

3. nginx vhost `sso.cloonar.com`

forceSSL, enableACME, acmeRoot = null — mirror the shape of the existing auth.cloonar.com vhost at hosts/web-arm/modules/authelia.nix:394-397, which is the correct pattern for this host's DNS-01 setup. Reverse-proxy to Keycloak on loopback with recommendedProxySettings.

SPNEGO-specific, and the most likely thing to get silently wrong: the Negotiate token travels in the Authorization request header and a Kerberos ticket can be several KB — far past nginx's default header buffers. Without enlarged large_client_header_buffers (and proxy buffer sizes to match), SPNEGO fails as a bare 400 Bad Request before it reaches Keycloak, which looks nothing like an auth problem. Size these deliberately and comment why.

4. Declarative realm via `services.keycloak.realmFiles`

A realm JSON checked into the repo, listed in realmFiles. Know the semantics before relying on them: the module symlinks the files into /run/keycloak/data/import/ and passes --import-realm, and Keycloak skips the import entirely if the realm already exists. It is create-if-absent, not converge — running on every start but a no-op after the first. During M3 that is fine (delete the realm to re-import; nothing is in it yet). Say so in a comment so nobody later assumes editing the JSON applies changes.

The realm must configure:

LDAP user federation against ldap.cloonar.com, usersDn = ou=users,dc=cloonar,dc=com, bind DN cn=linuxbind,ou=system,ou=users,dc=cloonar,dc=com, editMode = READ_ONLY, StartTLS. usernameLDAPAttribute must be uid — the LDAP uid is dominik.polakovics, which is exactly the short name of the Kerberos principal, and that correspondence is what lets Keycloak resolve a SPNEGO ticket to an LDAP user.
Kerberos integration on that federation provider: allowKerberosAuthentication = true, kerberosRealm = CLOONAR.COM, serverPrincipal = HTTP/sso.cloonar.com@CLOONAR.COM, keyTab = the runtime path from step 2.
useKerberosForPasswordAuthentication = false. This is the load-bearing scoping switch from ADR-0022 — with it off, Keycloak validates SPNEGO tickets but LDAP password logins still go to userPassword via an ordinary bind. Turning it on would make every LDAP login depend on the KDC, which is exactly what the ADR rejects.
The Kerberos execution in the browser flow set to ALTERNATIVE. A fresh realm ships it DISABLED, so SPNEGO silently never fires and everything falls through to the login form — the single most common cause of "Keycloak ignores my ticket". ALTERNATIVE (not REQUIRED) is also what preserves the fallback: users with no ticket, and every other tenant, still get the normal username/password form.
No clients. M4 owns those.

5. The LDAP bind credential

It cannot go in the realm JSON — realmFiles takes Nix store paths and those are world-readable.

Preferred: Keycloak's files-plaintext vault SPI (vault / vault-dir in services.keycloak.settings), with the realm JSON referencing ${vault.<key>} for bindCredential and sops materialising the file. Note the vault's filename convention is <realm>_<key>, and the file must be readable by the keycloak user. Reuse the existing linuxbind-password secret rather than adding a new one — set its path to land in the vault directory under the right name.

If the vault SPI turns out not to be wireable through services.keycloak.settings on this channel, fall back to entering the bind credential once in the admin console after first import, and document that clearly in the module and the PR as a known non-declarative element.

ADR amendment (required)

docs/adr/0022-keycloak-and-scoped-kerberos.md currently defers the declarative-vs-click-ops question to M4. That was wrong — LDAP federation and the Kerberos settings are realm configuration, so it lands at M3. Amend that section to record: realmFiles chosen, its create-if-absent semantics and what they cost, how the bind credential is handled, and that convergence (e.g. keycloak-config-cli) is revisited at M4 when real clients and sessions make "delete the realm to change it" unacceptable.

Out of scope

Any change to authelia.nix, auth.cloonar.com, or any existing vhost.
OIDC clients, oauth2-proxy, retiring anything.
hosts/mail/*, hosts/nb/*.
GSSAPI to Dovecot or SSH.

Verification

scripts/pre-commit dry-builds web-arm. Do not run scripts/test-configuration manually.
The agent cannot reach the fleet.

Human, after deploy:

Add DNS A/AAAA for sso.cloonar.com → web-arm. The host must match the SPN exactly; the browser derives HTTP/sso.cloonar.com from the URL.
systemctl status keycloak on web-arm; confirm the realm imported and the keytab decoded to the expected path with keycloak ownership.
Confirm auth.cloonar.com and an existing client (e.g. Grafana) still work — Authelia must be unaffected.
From the dominik.polakovics session on nb, on the VPN, with klist showing a TGT: browse to https://sso.cloonar.com/realms/<realm>/account.
- Pass: the account console loads with no password prompt.
- Fail: a login form appears. Check the browser flow's Kerberos execution is ALTERNATIVE, then nginx header buffer sizes, then keytab enctypes/kvno.
Negative control: from a browser with no TGT (or the local dominik session), the same URL must show the normal login form and accept the LDAP password — proving useKerberosForPasswordAuthentication = false left ordinary auth intact.

Definition of done

Keycloak reachable at sso.cloonar.com, realm imported declaratively from the repo, SPNEGO configured.
No diff to authelia.nix or any existing vhost; web-arm dry-builds clean.
ADR-0022 amended per above.
PR opened with Closes #<this issue>, and the PR description states plainly whether step 4 is expected to pass or whether anything was left non-declarative.

Part of #266 (M3 of M1–M6). Follows #267 (KDC on `mail`, complete) and #269 (SSSD + Kerberos login on `nb`, complete — desktop login yields a TGT in KCM). **M3 is the decision gate for the whole workstream.** Keycloak comes up at `sso.cloonar.com` *beside* Authelia, which keeps serving all of production untouched. The single question M3 answers is: does a `nb` desktop login get the user into Keycloak with no password prompt? If yes, M4–M6 proceed. If no, the workstream stops and nothing has been cut over. Design context: `docs/adr/0022-keycloak-and-scoped-kerberos.md`. ## Nothing is cut over in this issue - `hosts/web-arm/modules/authelia.nix` is **not touched**. `auth.cloonar.com` keeps serving every existing client. - **No OIDC clients** are registered in Keycloak (that is M4). - No oauth2-proxy, no changes to any existing nginx vhost, nothing retired. - Two IdPs running side by side is the deliberate design, not a transitional accident. ## Prerequisites already in place - `keycloak-db-password`, `keycloak-bootstrap-admin`, `keycloak-keytab-b64` are committed in `hosts/web-arm/secrets.yaml`. - The keytab holds `HTTP/sso.cloonar.com@CLOONAR.COM`, kvno 2, enctypes `aes256-cts-hmac-sha1-96` + `aes128-cts-hmac-sha1-96`. - LDAP read access: bind as `cn=linuxbind,ou=system,ou=users,dc=cloonar,dc=com`, whose password is already in `hosts/web-arm/secrets.yaml` as `linuxbind-password`. Federation is READ_ONLY, so no new system DN and no LDAP write is needed. - PostgreSQL already runs on `web-arm` (`hosts/web-arm/modules/postgresql.nix`). - Headroom is not a concern: `web-arm` measured 10 GB available of 15.5 GB. - DNS for `sso.cloonar.com` is a human step (below). It does **not** block the build: ACME here is DNS-01 via the Hetzner API (`utils/modules/lego/lego.nix:11`), so the certificate issues without an A record. ## Scope Create **`hosts/web-arm/modules/keycloak/`** (module + realm JSON) and import it from `hosts/web-arm/configuration.nix`. ### 1. The Keycloak service - `services.keycloak` with the PostgreSQL backend and `database.passwordFile` = the `keycloak-db-password` secret. - **`initialAdminPassword` must stay unset.** The module renders it into `Environment=` in the unit file, which is world-readable in the Nix store. Supply `KC_BOOTSTRAP_ADMIN_USERNAME` and `KC_BOOTSTRAP_ADMIN_PASSWORD` instead via `systemd.services.keycloak.serviceConfig.EnvironmentFile` pointed at the `keycloak-bootstrap-admin` secret. Leaving the option null means the module emits no competing `Environment=` lines. - Keycloak 26 hostname semantics: `settings.hostname` is the full external URL. HTTP-only on loopback behind nginx, with `proxy-headers = "xforwarded"` so it trusts the reverse proxy's `X-Forwarded-*`. ### 2. The keytab `keycloak-keytab-b64` is base64. Decode it to a stable runtime path owned by the `keycloak` user, mode 0400, in an `ExecStartPre`. `hosts/mail/modules/dovecot.nix:324` is the existing in-repo pattern for rendering a secret into a runtime file at service start. Do not write it anywhere in `/nix/store`. The realm JSON references this path, so it must be fixed and known at build time. ### 3. nginx vhost `sso.cloonar.com` `forceSSL`, `enableACME`, `acmeRoot = null` — mirror the shape of the existing `auth.cloonar.com` vhost at `hosts/web-arm/modules/authelia.nix:394-397`, which is the correct pattern for this host's DNS-01 setup. Reverse-proxy to Keycloak on loopback with `recommendedProxySettings`. **SPNEGO-specific, and the most likely thing to get silently wrong:** the `Negotiate` token travels in the `Authorization` request header and a Kerberos ticket can be several KB — far past nginx's default header buffers. Without enlarged `large_client_header_buffers` (and proxy buffer sizes to match), SPNEGO fails as a bare `400 Bad Request` before it reaches Keycloak, which looks nothing like an auth problem. Size these deliberately and comment why. ### 4. Declarative realm via `services.keycloak.realmFiles` A realm JSON checked into the repo, listed in `realmFiles`. **Know the semantics before relying on them:** the module symlinks the files into `/run/keycloak/data/import/` and passes `--import-realm`, and Keycloak *skips the import entirely if the realm already exists*. It is create-if-absent, not converge — running on every start but a no-op after the first. During M3 that is fine (delete the realm to re-import; nothing is in it yet). Say so in a comment so nobody later assumes editing the JSON applies changes. The realm must configure: - **LDAP user federation** against `ldap.cloonar.com`, `usersDn = ou=users,dc=cloonar,dc=com`, bind DN `cn=linuxbind,ou=system,ou=users,dc=cloonar,dc=com`, `editMode = READ_ONLY`, StartTLS. `usernameLDAPAttribute` must be `uid` — the LDAP `uid` is `dominik.polakovics`, which is exactly the short name of the Kerberos principal, and that correspondence is what lets Keycloak resolve a SPNEGO ticket to an LDAP user. - **Kerberos integration on that federation provider:** `allowKerberosAuthentication = true`, `kerberosRealm = CLOONAR.COM`, `serverPrincipal = HTTP/sso.cloonar.com@CLOONAR.COM`, `keyTab` = the runtime path from step 2. - **`useKerberosForPasswordAuthentication = false`.** This is the load-bearing scoping switch from ADR-0022 — with it off, Keycloak validates SPNEGO tickets but LDAP password logins still go to `userPassword` via an ordinary bind. Turning it on would make every LDAP login depend on the KDC, which is exactly what the ADR rejects. - **The `Kerberos` execution in the browser flow set to `ALTERNATIVE`.** A fresh realm ships it `DISABLED`, so SPNEGO silently never fires and everything falls through to the login form — the single most common cause of "Keycloak ignores my ticket". `ALTERNATIVE` (not `REQUIRED`) is also what preserves the fallback: users with no ticket, and every other tenant, still get the normal username/password form. - No clients. M4 owns those. ### 5. The LDAP bind credential It cannot go in the realm JSON — `realmFiles` takes Nix store paths and those are world-readable. Preferred: Keycloak's **files-plaintext vault SPI** (`vault` / `vault-dir` in `services.keycloak.settings`), with the realm JSON referencing `${vault.<key>}` for `bindCredential` and sops materialising the file. Note the vault's filename convention is `<realm>_<key>`, and the file must be readable by the `keycloak` user. Reuse the existing `linuxbind-password` secret rather than adding a new one — set its `path` to land in the vault directory under the right name. If the vault SPI turns out not to be wireable through `services.keycloak.settings` on this channel, fall back to entering the bind credential once in the admin console after first import, and **document that clearly in the module and the PR** as a known non-declarative element. ## ADR amendment (required) `docs/adr/0022-keycloak-and-scoped-kerberos.md` currently defers the declarative-vs-click-ops question to M4. That was wrong — LDAP federation and the Kerberos settings *are* realm configuration, so it lands at M3. Amend that section to record: `realmFiles` chosen, its create-if-absent semantics and what they cost, how the bind credential is handled, and that convergence (e.g. `keycloak-config-cli`) is revisited at M4 when real clients and sessions make "delete the realm to change it" unacceptable. ## Out of scope - Any change to `authelia.nix`, `auth.cloonar.com`, or any existing vhost. - OIDC clients, oauth2-proxy, retiring anything. - `hosts/mail/*`, `hosts/nb/*`. - GSSAPI to Dovecot or SSH. ## Verification - `scripts/pre-commit` dry-builds `web-arm`. Do not run `scripts/test-configuration` manually. - The agent cannot reach the fleet. Human, after deploy: 1. Add DNS A/AAAA for `sso.cloonar.com` → `web-arm`. The host must match the SPN exactly; the browser derives `HTTP/sso.cloonar.com` from the URL. 2. `systemctl status keycloak` on `web-arm`; confirm the realm imported and the keytab decoded to the expected path with `keycloak` ownership. 3. Confirm `auth.cloonar.com` and an existing client (e.g. Grafana) still work — **Authelia must be unaffected.** 4. From the `dominik.polakovics` session on `nb`, on the VPN, with `klist` showing a TGT: browse to `https://sso.cloonar.com/realms/<realm>/account`. - **Pass:** the account console loads with no password prompt. - **Fail:** a login form appears. Check the browser flow's Kerberos execution is `ALTERNATIVE`, then nginx header buffer sizes, then keytab enctypes/kvno. 5. Negative control: from a browser with no TGT (or the local `dominik` session), the same URL must show the normal login form and accept the LDAP password — proving `useKerberosForPasswordAuthentication = false` left ordinary auth intact. ## Definition of done - Keycloak reachable at `sso.cloonar.com`, realm imported declaratively from the repo, SPNEGO configured. - No diff to `authelia.nix` or any existing vhost; `web-arm` dry-builds clean. - ADR-0022 amended per above. - PR opened with `Closes #<this issue>`, and the PR description states plainly whether step 4 is expected to pass or whether anything was left non-declarative.

dominik.polakovics added the

ready-for-agent

label

2026-08-07 13:57:30 +02:00

dominik.polakovics referenced this issue from a commit

2026-08-07 15:01:38 +02:00

feat(web-arm): Keycloak at sso.cloonar.com beside Authelia, declarative realm + SPNEGO

dominik.polakovics referenced this issue from a pull request that will close it,

2026-08-07 15:02:21 +02:00

feat(web-arm): Keycloak at sso.cloonar.com beside Authelia, declarative realm + SPNEGO #272

dominik.polakovics closed this issue

2026-08-07 15:54:46 +02:00