deckle-hosted editor instances live and die with the control plane (unit comments, ADR-0032 consequence) #470

New issue

Closed

opened 2026-10-14 19:58:05 +02:00 by dominik.polakovics · 0 comments

dominik.polakovics commented

2026-10-14 19:58:05 +02:00

Owner

Summary

Cloonar/deckle-hosted#123 changes the editor-instance lifecycle: the control
plane stops every editor instance on SIGTERM and removes leftovers at startup.
Restart survival of instances is dropped. The unit in
hosts/deckle-01/modules/deckle-hosted/default.nix needs no functional
change, but its comments describe a different world, and ADR-0032 should
record the decision.

What was observed on deckle-01 on 2026-10-14

The container's processes run in the user manager's libpod-<id>.scope
under user@1500.service; only conmon and the control plane's
podman wait watchers sit in /system.slice/deckle-hosted.service.
No libpod-conmon-*.scope exists. The module comment "podman ... drives its
containers through the user's systemd instance" is half true: the
container, yes; conmon, no.
On systemctl stop deckle-hosted the unit's SIGTERM reaches conmon, which
forwards it to the container. The harness exits within ~30 ms and conmon
then runs the --rm cleanup. When the control plane's main process exits,
systemd's final-SIGTERM phase signals the remaining cgroup again; if that
hits the cleanup child first, the container stays in storage as exited.
Happened at 18:33:34 (site suneko), did not happen at 13:12:25. Every later
cold start of the site failed on the container name until
podman rm deckle-instance-suneko was run as deckle.
The Delegate = true comment says the limits need a cgroup subtree the
user may write. The instance limits are in fact applied on scopes of the
user manager; whether podman build needs Delegate is unverified. Leave
the setting, fix the comment or mark it unverified.

Desired changes

Module comments: state that the control plane stops its instances on
SIGTERM within a bound well under TimeoutStopSec (default 90 s; the
manager's bound is a code constant in deckle-hosted, see #123), that
KillMode=control-group is the backstop for anything left, and that a
deploy therefore interrupts open editors for a few seconds (they restart
on the next request).
Decide whether to set TimeoutStopSec explicitly with that comment, or
rely on the default.
ADR-0032: add a consequence, or write a new ADR, recording that
deckle-hosted editor instances live and die with the control plane and
why: survival never held under the system unit, every deploy so far
changed the harness hash, which makes the control plane stop old
instances anyway, and keeping conmon out of the unit's cgroup
(systemd-run --user --scope) would add a second lifecycle authority for
a benefit only a control-plane-only restart shows.
Optional follow-up: find out why podman 5.8.8 does not create its
libpod-conmon-<id>.scope here; not needed for the decision above.

Verification

After deckle-hosted#123 is deployed: restart the unit while an editor is
open; the journal shows the control plane stopping the instance before
"shutdown complete", and podman ps -a --filter label=deckle-hosted.role=instance as deckle is empty.

Out of scope

Any change to the deckle-hosted code (#123, #124 there).

## Summary Cloonar/deckle-hosted#123 changes the editor-instance lifecycle: the control plane stops every editor instance on SIGTERM and removes leftovers at startup. Restart survival of instances is dropped. The unit in `hosts/deckle-01/modules/deckle-hosted/default.nix` needs no functional change, but its comments describe a different world, and ADR-0032 should record the decision. ## What was observed on deckle-01 on 2026-10-14 - The container's processes run in the user manager's `libpod-<id>.scope` under `user@1500.service`; only conmon and the control plane's `podman wait` watchers sit in `/system.slice/deckle-hosted.service`. No `libpod-conmon-*.scope` exists. The module comment "podman ... drives its containers through the user's systemd instance" is half true: the container, yes; conmon, no. - On `systemctl stop deckle-hosted` the unit's SIGTERM reaches conmon, which forwards it to the container. The harness exits within ~30 ms and conmon then runs the `--rm` cleanup. When the control plane's main process exits, systemd's final-SIGTERM phase signals the remaining cgroup again; if that hits the cleanup child first, the container stays in storage as exited. Happened at 18:33:34 (site suneko), did not happen at 13:12:25. Every later cold start of the site failed on the container name until `podman rm deckle-instance-suneko` was run as `deckle`. - The `Delegate = true` comment says the limits need a cgroup subtree the user may write. The instance limits are in fact applied on scopes of the user manager; whether `podman build` needs Delegate is unverified. Leave the setting, fix the comment or mark it unverified. ## Desired changes - Module comments: state that the control plane stops its instances on SIGTERM within a bound well under `TimeoutStopSec` (default 90 s; the manager's bound is a code constant in deckle-hosted, see #123), that `KillMode=control-group` is the backstop for anything left, and that a deploy therefore interrupts open editors for a few seconds (they restart on the next request). - Decide whether to set `TimeoutStopSec` explicitly with that comment, or rely on the default. - ADR-0032: add a consequence, or write a new ADR, recording that deckle-hosted editor instances live and die with the control plane and why: survival never held under the system unit, every deploy so far changed the harness hash, which makes the control plane stop old instances anyway, and keeping conmon out of the unit's cgroup (`systemd-run --user --scope`) would add a second lifecycle authority for a benefit only a control-plane-only restart shows. - Optional follow-up: find out why podman 5.8.8 does not create its `libpod-conmon-<id>.scope` here; not needed for the decision above. ## Verification After deckle-hosted#123 is deployed: restart the unit while an editor is open; the journal shows the control plane stopping the instance before "shutdown complete", and `podman ps -a --filter label=deckle-hosted.role=instance` as `deckle` is empty. ## Out of scope Any change to the deckle-hosted code (#123, #124 there).

dominik.polakovics added the

ready-for-agent

enhancement

labels

2026-10-14 19:58:05 +02:00

dominik.polakovics referenced this issue from a pull request that will close it,

2026-10-18 17:21:45 +02:00

docs(deckle-01): editor instances live and die with the control plane #499

dominik.polakovics referenced this issue

2026-10-18 17:24:25 +02:00

docs(deckle-01): editor instances live and die with the control plane #499

dominik.polakovics closed this issue