K3s + CloudNativePG cluster setup complete

memory/2026-02-18.md

@@ -67,3 +67,77 @@
 - Firewall: ID 10553199
 - Load Balancer: ID 5833603, IP 46.225.37.146
 - SSH keys: dominik-nb01 (ID 107656266), openclaw-vm (ID 107656268)
+
+## Coolify 3-Node HA Setup Complete
+
+### Infrastructure
+- **coolify-mgr** (188.34.201.101, 10.0.1.1) — Coolify UI + etcd
+- **coolify-w1** (46.225.62.90, 10.0.1.2) — Apps + etcd + Patroni PRIMARY + PgBouncer
+- **coolify-w2** (46.224.208.205, 10.0.1.4) — Apps + etcd + Patroni REPLICA + PgBouncer
+- Hetzner server ID for w2: 121361614, Coolify UUID: mwccg08sokosk4wgw40g08ok
+
+### Components
+- **etcd 3.5.17** on all 3 nodes (quay.io/coreos/etcd, ARM64 compatible)
+- **Patroni + PostgreSQL 16** on workers (custom Docker image `patroni:local`)
+- **PgBouncer** (edoburu/pgbouncer) on workers — routes to current primary
+- **Watcher** (systemd timer, every 5s) updates PgBouncer config on failover
+
+### Key Facts
+- Docker daemon.json on all nodes: `172.17.0.0/12` pool (fixes 10.0.x conflict with Hetzner private net)
+- Infra compose: `/opt/infra/docker-compose.yml` on each node
+- Patroni config: `/opt/infra/patroni/patroni.yml`
+- PgBouncer config: `/opt/infra/pgbouncer/pgbouncer.ini`
+- Watcher script: `/opt/infra/pgbouncer/update-primary.sh`
+- Failover log: `/opt/infra/pgbouncer/failover.log`
+- `docfast` database created and replicated
+- Failover tested: pg1→pg2 promotion + pg1 rejoin as replica ✅
+- Switchover tested: pg2→pg1 clean switchover ✅
+- Cost: €11.67/mo (3x CAX11)
+
+### Remaining Steps
+- [ ] Migrate DocFast data from 167.235.156.214 to Patroni cluster
+- [ ] Deploy DocFast app via Coolify on both workers
+- [ ] Set up BorgBackup on new nodes
+- [ ] Add docfast user SCRAM hash to PgBouncer userlist
+- [ ] Create project-scoped API tokens for CEO agents
+
+## K3s + CloudNativePG Setup Complete
+
+### Architecture
+- **k3s-mgr** (188.34.201.101, 10.0.1.5) — K3s control plane, Hetzner ID 121365837
+- **k3s-w1** (159.69.23.121, 10.0.1.6) — Worker, Hetzner ID 121365839
+- **k3s-w2** (46.225.169.60, 10.0.1.7) — Worker, Hetzner ID 121365840
+
+### Cluster Components
+- K3s v1.34.4 (Traefik DaemonSet on workers, servicelb disabled)
+- CloudNativePG 1.25.1 (operator in cnpg-system namespace)
+- cert-manager 1.17.2 (Let's Encrypt ClusterIssuer)
+- PostgreSQL 17.4 (CNPG managed, 2 instances, 1 primary + 1 replica)
+- PgBouncer Pooler (CNPG managed, 2 instances, transaction mode)
+
+### Namespaces
+- postgres: CNPG cluster + pooler
+- docfast: DocFast app deployment
+- cnpg-system: CNPG operator
+- cert-manager: Certificate management
+
+### DocFast Deployment
+- 2 replicas, one per worker
+- Image: docker.io/library/docfast:latest (locally built + imported via k3s ctr)
+- DB: main-db-pooler.postgres.svc:5432
+- Health: /health on port 3100
+- 53 API keys migrated from old server
+
+### Key Learnings
+- Docker images must be imported with `k3s ctr images import --all-platforms` (not `ctr -n k3s.io`)
+- CNPG tolerations field caused infinite restart loop — removed to fix
+- DB table ownership must be set to app user after pg_restore with --no-owner
+
+### Remaining
+- [ ] Switch DNS docfast.dev → worker IP (159.69.23.121 or 46.225.169.60)
+- [ ] TLS cert will auto-complete after DNS switch
+- [ ] Update Stripe webhook endpoint if needed
+- [ ] Set up CI/CD pipeline for automated deploys
+- [ ] Create CEO namespace RBAC
+- [ ] Decommission old server (167.235.156.214)
+- [ ] Clean up Docker from workers (only needed containerd/K3s)