M0: make the KDC survivable before it becomes the only credential #302

New issue

Open

opened 2026-08-08 13:30:16 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-08-08 13:30:16 +02:00

Owner

M0 of #299 — blocks everything else in that epic.

Today a KDC loss costs nb its SSO and nothing more, because LDAP userPassword still authenticates everyone everywhere. After #299 locks it to {CRYPT}! there is no fallback credential anywhere, and an outage costs all web SSO, all identity-tenant mail, and all fresh server logins. Laptops keep working from SSSD's cache — the wrong subset: the humans are fine and the services are down.

A replica (kprop/iprop) was considered and deliberately deferred. That is defensible, but it makes the two items below prerequisites rather than hygiene.

1. The KDC backup is probably not restorable

Borg sweeps /var and does not exclude /var/lib/krb5kdc (utils/modules/borgbackup.nix:60-85), so principal and the .k5.CLOONAR.COM stash are captured — but captured live, while the KDC holds them open. MIT's documented backup method is kdb5_util dump. A file-level copy of an open database may restore cleanly or may not, and "may" is not a recovery plan when it is the only copy.

Timer dumping to a root-only file that borg then picks up. The dump carries key material (master-key-encrypted, but still) — 0600, root.
Then actually restore it somewhere and prove a kinit succeeds against the result. A backup nobody has restored is not a backup, and under a single-KDC design it is the entire recovery story.
Write down the resulting RTO honestly: how long from "mail is gone" to "people can log in".

2. Break-glass

A locally declared admin account with a real password on mail and web-arm. With a replica this is belt-and-braces; without one it is the only brace. It is also what covers the case the backup cannot: a KDB restore that fails.

Definition of done

kdb5_util dump on a timer, output covered by borg and not world-readable.
A restore performed at least once, with the kinit against it recorded in the PR.
Break-glass accounts on both hosts, and where their passwords live written down somewhere that does not require Kerberos to reach.

M0 of #299 — blocks everything else in that epic. Today a KDC loss costs `nb` its SSO and nothing more, because LDAP `userPassword` still authenticates everyone everywhere. After #299 locks it to `{CRYPT}!` there is **no fallback credential anywhere**, and an outage costs all web SSO, all identity-tenant mail, and all fresh server logins. Laptops keep working from SSSD's cache — the wrong subset: the humans are fine and the services are down. A replica (`kprop`/`iprop`) was considered and deliberately deferred. That is defensible, but it makes the two items below prerequisites rather than hygiene. ## 1. The KDC backup is probably not restorable Borg sweeps `/var` and does not exclude `/var/lib/krb5kdc` (`utils/modules/borgbackup.nix:60-85`), so `principal` and the `.k5.CLOONAR.COM` stash are captured — but captured **live**, while the KDC holds them open. MIT's documented backup method is `kdb5_util dump`. A file-level copy of an open database may restore cleanly or may not, and "may" is not a recovery plan when it is the only copy. - Timer dumping to a root-only file that borg then picks up. The dump carries key material (master-key-encrypted, but still) — `0600`, root. - Then **actually restore it** somewhere and prove a `kinit` succeeds against the result. A backup nobody has restored is not a backup, and under a single-KDC design it is the entire recovery story. - Write down the resulting RTO honestly: how long from "`mail` is gone" to "people can log in". ## 2. Break-glass A locally declared admin account with a real password on `mail` and `web-arm`. With a replica this is belt-and-braces; without one it is the only brace. It is also what covers the case the backup cannot: a KDB restore that fails. ## Definition of done - `kdb5_util dump` on a timer, output covered by borg and not world-readable. - A restore performed at least once, with the `kinit` against it recorded in the PR. - Break-glass accounts on both hosts, and where their passwords live written down somewhere that does not require Kerberos to reach.

dominik.polakovics added the

ready-for-human

label

2026-08-08 13:30:16 +02:00

dominik.polakovics referenced this issue

2026-08-09 01:47:41 +02:00

Kerberos becomes the authoritative password store for the identity tenant #299

dominik.polakovics added the

enhancement

label

2026-08-09 01:47:52 +02:00

dominik.polakovics referenced this issue