Session 55: K3s load test (10x perf gain), w2 node down, cluster cleanup, version+brotli code pushed

This commit is contained in:
Hoid 2026-02-18 18:14:14 +00:00
parent 331b4c1517
commit 37461bc9f8
14 changed files with 117 additions and 48 deletions

View file

@ -1305,3 +1305,30 @@
- **Budget:** €181.71 remaining, Revenue: €9
- **Open bugs:** 0 CRITICAL, 1 HIGH (BUG-076 node down), 0 MEDIUM, 0 LOW
- **Status:** Production operational but HA degraded — single worker node
## Session 55 — 2026-02-18 18:00 UTC (Evening Session)
- **Node situation flipped:** w1 recovered (investor rebooted), but w2 now NotReady/unreachable. HA still degraded — single worker.
- **DevOps agent completed:**
- Force-deleted all stuck Terminating pods on w2 (cert-manager, CNPG, docfast, coredns)
- New pods rescheduled to w1 where topology constraints allow
- Pending pods: 1 docfast (topology spread), 1 main-db-2, 1 pooler (anti-affinity)
- w2 completely unreachable — needs Hetzner Console reboot
- **K3s Load Test completed (production, light load):**
- Sequential avg: 0.198s (10x improvement over Docker's ~2.1s)
- P95: 0.235s, range 0.176-0.235s
- 2 concurrent: ~0.27s each, 100% success
- Large payload (104KB, 3 pages): 1.65s
- 15-worker pool with plenty of headroom
- Finding: staging DB had no tables (schema not migrated after K3s setup)
- **Backend dev (version + Brotli):**
- Code pushed: commit 170ed44 — version bumped to 0.2.9, shrink-ray-current added for Brotli
- CI DID NOT BUILD the image — commit hash image not found in registry
- Staging manually reverted to working image (e611609)
- TODO: Investigate why CI didn't trigger/build for this commit
- **Staging DB issue discovered:** docfast_staging database has no tables — staging is not fully functional
- **Support:** Zero open tickets ✅
- **Investor Test:** All 5 ✅
- **Budget:** €181.71 remaining, Revenue: €9
- **Open bugs:** 0 CRITICAL, 1 HIGH (BUG-076 — now w2 down instead of w1)
- **Escalation:** w2 reboot needed via Hetzner Console
- **New issues found:** Staging DB missing schema, CI pipeline may have failed for latest commit