docs: record the web-arm Hetzner egress loss incident #388

Open

dominik.polakovics wants to merge 1 commit from docs/web-arm-hetzner-egress-loss into main

dominik.polakovics commented

2026-08-24 23:48:52 +02:00

Owner

Immich playback from home stalls for minutes at a time while speedtests look fine. Root cause is outside our infrastructure: Hetzner intermittently drops traffic from web-arm's segment toward AS25255 (Drei Austria), in the outbound direction only.

This PR records the incident so the evidence survives the conversation — it is documentation only, no config changes.

ticket-hetzner.md — the support request for Hetzner (German), ready to send
README.md — incident write-up, method, and what was ruled out
data/ — the raw measurement logs the ticket cites

The decisive evidence

Three transfers started in the same instant, the Cloudflare one leaving the same interface on the same host:

window	web-arm → home	amzebs-01 → home	web-arm → Cloudflare
r2	2 MB	400 MB	59 MB/s
r5	2 MB	400 MB	73 MB/s
r8	1 MB	216 MB	51 MB/s

web-arm sustains 43-75 MB/s to Cloudflare while its transfer home is dead, with tx_dropped/tx_errors at 0 — so its NIC, transmit queue and hypervisor are healthy. amzebs-01 reaches the same destination cleanly over a path identical from hop 5 onward; the two differ only in the first hop out of the server (162.55.117.224 vs 49.12.28.81).

fw is cleared by its own counters: 1524 packets received from web-arm, 1524 forwarded. It cannot forward what never arrives.

Why the method notes are in the write-up

Two traps each produced a wrong conclusion before the harness was tightened, and both are cheap to fall into again:

the fault is episodic, so measurements taken minutes apart are incomparable — every comparison that matters must run simultaneously
the symptom lives in the download direction; probing home → web-arm shows a clean path and leads nowhere

The write-up also documents that ICMP is unusable on this path — the 8-17 % loss it reports persists in windows where TCP moves 400 MB at 11 MB/s, which is impossible at that real loss rate, so the ticket disclaims it rather than leaning on it.

Not done

The wg_de_exit detour that would route around the affected segment is described but not deployed. The fault is Hetzner's to fix; deploying a workaround is a separate decision.

Immich playback from home stalls for minutes at a time while speedtests look fine. Root cause is outside our infrastructure: **Hetzner intermittently drops traffic from web-arm's segment toward AS25255 (Drei Austria)**, in the outbound direction only. This PR records the incident so the evidence survives the conversation — it is documentation only, no config changes. ### Contents - `ticket-hetzner.md` — the support request for Hetzner (German), ready to send - `README.md` — incident write-up, method, and what was ruled out - `data/` — the raw measurement logs the ticket cites ### The decisive evidence Three transfers started in the same instant, the Cloudflare one leaving the same interface on the same host: | window | web-arm → home | amzebs-01 → home | web-arm → Cloudflare | |--------|---------------|------------------|----------------------| | r2 | 2 MB | 400 MB | 59 MB/s | | r5 | 2 MB | 400 MB | 73 MB/s | | r8 | 1 MB | 216 MB | 51 MB/s | web-arm sustains 43-75 MB/s to Cloudflare while its transfer home is dead, with `tx_dropped`/`tx_errors` at 0 — so its NIC, transmit queue and hypervisor are healthy. `amzebs-01` reaches the same destination cleanly over a path identical from hop 5 onward; the two differ only in the first hop out of the server (`162.55.117.224` vs `49.12.28.81`). fw is cleared by its own counters: 1524 packets received from web-arm, 1524 forwarded. It cannot forward what never arrives. ### Why the method notes are in the write-up Two traps each produced a wrong conclusion before the harness was tightened, and both are cheap to fall into again: - the fault is **episodic**, so measurements taken minutes apart are incomparable — every comparison that matters must run simultaneously - the symptom lives in the **download** direction; probing `home → web-arm` shows a clean path and leads nowhere The write-up also documents that ICMP is unusable on this path — the 8-17 % loss it reports persists in windows where TCP moves 400 MB at 11 MB/s, which is impossible at that real loss rate, so the ticket disclaims it rather than leaning on it. ### Not done The `wg_de_exit` detour that would route around the affected segment is described but not deployed. The fault is Hetzner's to fix; deploying a workaround is a separate decision.

dominik.polakovics added 1 commit

2026-08-24 23:48:52 +02:00

docs: record the web-arm Hetzner egress loss incident 214d80fd3c

Immich playback from home stalls because Hetzner intermittently drops traffic
from web-arm's segment toward AS25255 (Drei Austria), in the outbound direction
only. Adds the support request filed with Hetzner plus the raw measurements it
cites.

The decisive evidence is three simultaneous transfers: during a collapse
web-arm sustains 43-75 MB/s to Cloudflare on the same interface while its
transfer home is dead at 1-2 MB, and amzebs-01 reaches the same destination
cleanly over a path identical from hop 5 onward.

Two method notes are written up because both produced wrong conclusions first:
the fault is episodic, so comparisons must run simultaneously, and the symptom
lives in the download direction, not the upload.

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin docs/web-arm-hetzner-egress-loss:docs/web-arm-hetzner-egress-loss

git switch docs/web-arm-hetzner-egress-loss

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff docs/web-arm-hetzner-egress-loss

git switch docs/web-arm-hetzner-egress-loss

git rebase main

git switch main

git merge --ff-only docs/web-arm-hetzner-egress-loss

git switch docs/web-arm-hetzner-egress-loss

git rebase main

git switch main

git merge --no-ff docs/web-arm-hetzner-egress-loss

git switch main

git merge --squash docs/web-arm-hetzner-egress-loss