feat(web-arm): install paperless-ngx #385

New issue

Open

opened 2026-08-23 12:40:17 +02:00 by dominik.polakovics · 1 comment

dominik.polakovics commented

2026-08-23 12:40:17 +02:00

Owner

This was generated by AI during triage.

Design settled in a grilling session on 2026-08-23; the Agent Brief comment below is the contract.

Problem Statement

Household documents (invoices, contracts, tax records, scanned mail) have no searchable, structured archive. They live scattered across mailboxes, scanner output folders and ad-hoc storage, and several people in the household need to keep their documents private from each other.

Solution

Deploy paperless-ngx on web-arm as a native NixOS service, publicly reachable at paperless.cloonar.com, authenticated via the existing Keycloak SSO with a group gate, with real multi-user separation inside the app. Documents arrive via the web UI/mobile apps and via scan-to-email to a dedicated mailbox; media is stored on a dedicated Hetzner Storage-Box CIFS share; backups combine Storage-Box snapshots, the existing PostgreSQL dumps, and a nightly paperless-native document export picked up by borg.

User Stories

As a household member, I want to upload documents from browser or mobile app and have them OCR'd (German + English) and full-text searchable, so that I can find any document later.
As a household member, I want to scan-to-email to scan+<user>@cloonar.com and have the document land in my own archive, so that paper mail is archived without touching a computer.
As a household member, I want my documents invisible to other users, so that private records stay private.
As the admin, I want login restricted to members of a Keycloak paperless group, so that other realm accounts cannot even authenticate.
As the admin, I want a single local-password break-glass account, so that an SSO outage does not lock me out.
As the admin, I want the archive protected by snapshots plus a restore-anywhere export in borg, so that irreplaceable documents survive deletion, ransomware or provider failure.

Implementation Decisions

Native NixOS module (services.paperless, NixOS 26.05) in a new web-arm host module, imported from the host configuration. No containers.
Auth: OIDC against the existing Keycloak (sso.cloonar.com) via paperless's django-allauth support. The OIDC client secret is injected with a sops template so it never enters the Nix store, with a restart trigger on the rendered config hash — the same pattern the Immich module uses. Local password login stays enabled for exactly one break-glass admin account.
Keycloak (declarative realm config): add a paperless confidential client, a /paperless group carrying a policy-paperless realm role, and a role-gated browser flow mirroring the existing Immich-gated flow, plus the client-secret vault wiring the other clients use.
Multi-user separation: OIDC auto-signup on, but new accounts arrive permissionless until the admin grants access; per-document ownership enforces separation inside the app.
Ingestion: web UI / mobile apps (owner = uploader); scan-to-email to a new scan@cloonar.com mailbox using plus-addressing (scan+alice@), with per-user mail rules and owner-assigning workflows configured later in the UI. The consume directory stays local-only for admin bulk imports over SSH. No network share.
Storage: media tree (originals + archive + thumbnails) on a new dedicated Hetzner Storage-Box sub-account mounted via CIFS; data dir (index, classifier) on the local SSD. The mount replicates the post-fix Immich mount semantics exactly: persistent mount (never an idle-timeout automount), nofail, generous mount timeout, ownership forced to the paperless user, and RequiresMountsFor on the paperless units. Chosen eyes-open: browsing latency and a hard mount dependency are accepted.
Database: the existing shared PostgreSQL 14 instance, covered by the existing postgresqlBackup dumps. Dedicated Redis instance for paperless on a unix socket.
Backups: CIFS media excluded from borg (the borg repo sits on the same Storage Box); Hetzner snapshots enabled on the sub-account (human action); nightly document_exporter run into a local directory that the existing borg /var paths already cover.
Processing: OCR languages deu+eng; PDFs/images only — no Tika/Gotenberg.
Resource etiquette: CPUWeight 20 on the paperless worker units and a low task-worker count so OCR never starves the websites sharing the box (Immich precedent).
Exposure: public nginx vhost paperless.cloonar.com with ACME via the existing lego setup; PAPERLESS_URL set to the exact external URL for CSRF.

Testing Decisions

This repo has no unit tests; the gate is the pre-commit hook's dry-build of affected hosts (scripts/test-configuration via the installed hook — commit and let the hook run it, don't invoke it manually). Runtime verification happens after deploy by the maintainer.

Out of Scope

Creating the scan@cloonar.com mailbox on the mail host, the DNS record for paperless.cloonar.com, and the Storage-Box sub-account + snapshot schedule — human actions; the PR must list them.
Secret values — the agent never edits sops files; the PR must name every secret the maintainer needs to add and where.
Runtime UI configuration: IMAP account, per-user mail rules/workflows, user permissions, Keycloak/LDAP group membership.
Office-document support (Tika/Gotenberg), any network consume share (SMB/SFTP/WireGuard), and Authelia forward-auth in front of the vhost.

Blocked by

None - can start immediately

> *This was generated by AI during triage.* Design settled in a grilling session on 2026-08-23; the Agent Brief comment below is the contract. ## Problem Statement Household documents (invoices, contracts, tax records, scanned mail) have no searchable, structured archive. They live scattered across mailboxes, scanner output folders and ad-hoc storage, and several people in the household need to keep their documents private from each other. ## Solution Deploy paperless-ngx on web-arm as a native NixOS service, publicly reachable at `paperless.cloonar.com`, authenticated via the existing Keycloak SSO with a group gate, with real multi-user separation inside the app. Documents arrive via the web UI/mobile apps and via scan-to-email to a dedicated mailbox; media is stored on a dedicated Hetzner Storage-Box CIFS share; backups combine Storage-Box snapshots, the existing PostgreSQL dumps, and a nightly paperless-native document export picked up by borg. ## User Stories 1. As a household member, I want to upload documents from browser or mobile app and have them OCR'd (German + English) and full-text searchable, so that I can find any document later. 2. As a household member, I want to scan-to-email to `scan+<user>@cloonar.com` and have the document land in my own archive, so that paper mail is archived without touching a computer. 3. As a household member, I want my documents invisible to other users, so that private records stay private. 4. As the admin, I want login restricted to members of a Keycloak `paperless` group, so that other realm accounts cannot even authenticate. 5. As the admin, I want a single local-password break-glass account, so that an SSO outage does not lock me out. 6. As the admin, I want the archive protected by snapshots plus a restore-anywhere export in borg, so that irreplaceable documents survive deletion, ransomware or provider failure. ## Implementation Decisions - **Native NixOS module** (`services.paperless`, NixOS 26.05) in a new web-arm host module, imported from the host configuration. No containers. - **Auth:** OIDC against the existing Keycloak (`sso.cloonar.com`) via paperless's django-allauth support. The OIDC client secret is injected with a sops template so it never enters the Nix store, with a restart trigger on the rendered config hash — the same pattern the Immich module uses. Local password login stays enabled for exactly one break-glass admin account. - **Keycloak (declarative realm config):** add a `paperless` confidential client, a `/paperless` group carrying a `policy-paperless` realm role, and a role-gated browser flow mirroring the existing Immich-gated flow, plus the client-secret vault wiring the other clients use. - **Multi-user separation:** OIDC auto-signup on, but new accounts arrive permissionless until the admin grants access; per-document ownership enforces separation inside the app. - **Ingestion:** web UI / mobile apps (owner = uploader); scan-to-email to a new `scan@cloonar.com` mailbox using plus-addressing (`scan+alice@`), with per-user mail rules and owner-assigning workflows configured later in the UI. The consume directory stays local-only for admin bulk imports over SSH. No network share. - **Storage:** media tree (originals + archive + thumbnails) on a **new dedicated Hetzner Storage-Box sub-account** mounted via CIFS; data dir (index, classifier) on the local SSD. The mount replicates the post-fix Immich mount semantics exactly: persistent mount (never an idle-timeout automount), `nofail`, generous mount timeout, ownership forced to the paperless user, and `RequiresMountsFor` on the paperless units. Chosen eyes-open: browsing latency and a hard mount dependency are accepted. - **Database:** the existing shared PostgreSQL 14 instance, covered by the existing `postgresqlBackup` dumps. Dedicated Redis instance for paperless on a unix socket. - **Backups:** CIFS media excluded from borg (the borg repo sits on the same Storage Box); Hetzner snapshots enabled on the sub-account (human action); nightly `document_exporter` run into a local directory that the existing borg `/var` paths already cover. - **Processing:** OCR languages `deu+eng`; PDFs/images only — no Tika/Gotenberg. - **Resource etiquette:** CPUWeight 20 on the paperless worker units and a low task-worker count so OCR never starves the websites sharing the box (Immich precedent). - **Exposure:** public nginx vhost `paperless.cloonar.com` with ACME via the existing lego setup; `PAPERLESS_URL` set to the exact external URL for CSRF. ## Testing Decisions This repo has no unit tests; the gate is the pre-commit hook's dry-build of affected hosts (`scripts/test-configuration` via the installed hook — commit and let the hook run it, don't invoke it manually). Runtime verification happens after deploy by the maintainer. ## Out of Scope - Creating the `scan@cloonar.com` mailbox on the mail host, the DNS record for `paperless.cloonar.com`, and the Storage-Box sub-account + snapshot schedule — human actions; the PR must list them. - Secret *values* — the agent never edits sops files; the PR must name every secret the maintainer needs to add and where. - Runtime UI configuration: IMAP account, per-user mail rules/workflows, user permissions, Keycloak/LDAP group membership. - Office-document support (Tika/Gotenberg), any network consume share (SMB/SFTP/WireGuard), and Authelia forward-auth in front of the vhost. ## Blocked by None - can start immediately

dominik.polakovics commented

2026-08-23 12:40:22 +02:00

Author

Owner

This was generated by AI during triage.

Agent Brief

Category: enhancement
Summary: Add paperless-ngx to the web-arm host as a native NixOS service with Keycloak-gated OIDC login, CIFS-backed media storage, and a paperless-native export feeding the existing borg backups.

Current behavior:
web-arm runs nginx (ACME via lego), a shared PostgreSQL 14 with nightly dumps, sops-managed secrets, borg backups of /var to a Hetzner Storage Box, a declaratively-managed Keycloak realm (sso.cloonar.com), and several native-module services — Immich being the closest precedent (OIDC client secret via sops template, gated Keycloak browser flow, CIFS Storage-Box mount, CPUWeight throttling). There is no document management system.

Desired behavior:
https://paperless.cloonar.com serves paperless-ngx. Members of the Keycloak /paperless group log in via SSO; anyone else cannot complete authentication. One local-password admin account remains as break-glass. Uploaded/emailed documents are OCR'd in German and English and stored with the media tree on a dedicated Storage-Box CIFS share; the search index and classifier stay on local disk. A nightly document_exporter run writes a restore-anywhere export into a local directory that the existing borg job already covers.

Key interfaces:

services.paperless (NixOS 26.05) — enable; bind to loopback behind nginx; database on the existing shared PostgreSQL 14 instance (socket auth, DB provisioned the way other services on this host do it); a dedicated Redis instance on a unix socket; mediaDir on the CIFS mount, data dir on local disk; settings for PAPERLESS_URL = https://paperless.cloonar.com, PAPERLESS_OCR_LANGUAGE = deu+eng, and OIDC via the allauth PAPERLESS_SOCIALACCOUNT_PROVIDERS mechanism.
Secret handling — the OIDC client secret must never enter the Nix store: render whatever file carries it with a sops template, restart the paperless services on secret rotation via restartUnits, and add a restart trigger keyed on a hash of the rendered template content (mirror the Immich module's approach).
CIFS mount — a filesystem entry for the media path on a new Storage-Box sub-account, copying the post-fix Immich mount semantics exactly: plain persistent mount (never an idle-timeout automount), nofail, x-systemd.mount-timeout=2min, credentials from a sops secret, ownership forced to the paperless user/group via uid=/gid=, and RequiresMountsFor on every paperless unit that touches media. Do not introduce an automount or idle timeout — that exact construct caused multi-hour Immich outages.
Keycloak realm config (declarative JSON + module wiring) — add a confidential paperless client (secret from the file vault, redirect URI for the allauth OIDC callback on paperless.cloonar.com), a /paperless group carrying a policy-paperless realm role, a role-gated browser flow mirroring the existing browser-spnego immich flow family, and the client-secret plumbing the other clients use (the plainClients list plus a clientSecret-style sops declaration reading the same secret under a second name).
nginx vhost — paperless.cloonar.com with enableACME, forceSSL, acmeRoot = null, proxy to loopback with websocket support and a raised client_max_body_size for large scans.
borg job — exclude the CIFS media path from the default job's paths; the nightly export target directory must live under a path borg already covers.
Export timer — a systemd service+timer running paperless's document_exporter into that local directory, running as the paperless user, off-peak.
Resource etiquette — CPUWeight = 20 on the paperless worker/scheduler units and a low PAPERLESS_TASK_WORKERS so OCR never starves the websites sharing the box.

Acceptance criteria:

A new web-arm module (imported from the host configuration) enables paperless-ngx per the interfaces above; no container, no channel change, system.stateVersion untouched.
The OIDC client secret appears nowhere in the Nix store; rotation restarts the paperless services; config edits to the rendered file also trigger a restart.
The Keycloak realm JSON gains the paperless client, group, role and gated flow such that a realm user outside /paperless cannot complete login, mirroring the Immich gate's structure.
The CIFS mount is persistent with nofail and a generous timeout, and stopping the mount stops the paperless services (RequiresMountsFor) — with no automount/idle-timeout anywhere.
Media path excluded from borg; nightly document_exporter timer writes into a borg-covered local directory.
OCR is deu+eng; Tika/Gotenberg are absent from the closure.
Paperless worker units carry CPUWeight 20 and a task-worker count of 1.
The commit passes the pre-commit dry-build for web-arm (commit and let the installed hook run it; do not run the test script manually).
A PR is opened (never a direct push to main), Conventional-Commits style, and its description lists every secret the maintainer must add to the sops files (names + which file) and every external action required before deploy: Storage-Box sub-account + snapshots, DNS record for paperless.cloonar.com, scan@cloonar.com mailbox, Keycloak group membership.

Out of scope:

Editing any *.sops.yaml secret values (name what's needed in the PR instead).
The mail-host mailbox, DNS record, and Storage-Box sub-account themselves (human actions).
Runtime UI configuration: IMAP account, mail rules, owner-assigning workflows, user permissions.
Office-document support (Tika/Gotenberg), any network consume share, Authelia forward-auth on the vhost.

> *This was generated by AI during triage.* ## Agent Brief **Category:** enhancement **Summary:** Add paperless-ngx to the web-arm host as a native NixOS service with Keycloak-gated OIDC login, CIFS-backed media storage, and a paperless-native export feeding the existing borg backups. **Current behavior:** web-arm runs nginx (ACME via lego), a shared PostgreSQL 14 with nightly dumps, sops-managed secrets, borg backups of `/var` to a Hetzner Storage Box, a declaratively-managed Keycloak realm (`sso.cloonar.com`), and several native-module services — Immich being the closest precedent (OIDC client secret via sops template, gated Keycloak browser flow, CIFS Storage-Box mount, CPUWeight throttling). There is no document management system. **Desired behavior:** `https://paperless.cloonar.com` serves paperless-ngx. Members of the Keycloak `/paperless` group log in via SSO; anyone else cannot complete authentication. One local-password admin account remains as break-glass. Uploaded/emailed documents are OCR'd in German and English and stored with the media tree on a dedicated Storage-Box CIFS share; the search index and classifier stay on local disk. A nightly `document_exporter` run writes a restore-anywhere export into a local directory that the existing borg job already covers. **Key interfaces:** - `services.paperless` (NixOS 26.05) — enable; bind to loopback behind nginx; database on the existing shared PostgreSQL 14 instance (socket auth, DB provisioned the way other services on this host do it); a dedicated Redis instance on a unix socket; `mediaDir` on the CIFS mount, data dir on local disk; settings for `PAPERLESS_URL = https://paperless.cloonar.com`, `PAPERLESS_OCR_LANGUAGE = deu+eng`, and OIDC via the allauth `PAPERLESS_SOCIALACCOUNT_PROVIDERS` mechanism. - **Secret handling** — the OIDC client secret must never enter the Nix store: render whatever file carries it with a sops template, restart the paperless services on secret rotation via `restartUnits`, and add a restart trigger keyed on a hash of the rendered template content (mirror the Immich module's approach). - **CIFS mount** — a filesystem entry for the media path on a new Storage-Box sub-account, copying the post-fix Immich mount semantics exactly: plain persistent mount (never an idle-timeout automount), `nofail`, `x-systemd.mount-timeout=2min`, credentials from a sops secret, ownership forced to the paperless user/group via `uid=`/`gid=`, and `RequiresMountsFor` on every paperless unit that touches media. Do not introduce an automount or idle timeout — that exact construct caused multi-hour Immich outages. - **Keycloak realm config** (declarative JSON + module wiring) — add a confidential `paperless` client (`secret` from the file vault, redirect URI for the allauth OIDC callback on paperless.cloonar.com), a `/paperless` group carrying a `policy-paperless` realm role, a role-gated browser flow mirroring the existing `browser-spnego immich` flow family, and the client-secret plumbing the other clients use (the `plainClients` list plus a `clientSecret`-style sops declaration reading the same secret under a second name). - **nginx vhost** — `paperless.cloonar.com` with `enableACME`, `forceSSL`, `acmeRoot = null`, proxy to loopback with websocket support and a raised `client_max_body_size` for large scans. - **borg job** — exclude the CIFS media path from the default job's paths; the nightly export target directory must live under a path borg already covers. - **Export timer** — a systemd service+timer running paperless's `document_exporter` into that local directory, running as the paperless user, off-peak. - **Resource etiquette** — `CPUWeight = 20` on the paperless worker/scheduler units and a low `PAPERLESS_TASK_WORKERS` so OCR never starves the websites sharing the box. **Acceptance criteria:** - [ ] A new web-arm module (imported from the host configuration) enables paperless-ngx per the interfaces above; no container, no channel change, `system.stateVersion` untouched. - [ ] The OIDC client secret appears nowhere in the Nix store; rotation restarts the paperless services; config edits to the rendered file also trigger a restart. - [ ] The Keycloak realm JSON gains the paperless client, group, role and gated flow such that a realm user outside `/paperless` cannot complete login, mirroring the Immich gate's structure. - [ ] The CIFS mount is persistent with `nofail` and a generous timeout, and stopping the mount stops the paperless services (RequiresMountsFor) — with no automount/idle-timeout anywhere. - [ ] Media path excluded from borg; nightly `document_exporter` timer writes into a borg-covered local directory. - [ ] OCR is `deu+eng`; Tika/Gotenberg are absent from the closure. - [ ] Paperless worker units carry CPUWeight 20 and a task-worker count of 1. - [ ] The commit passes the pre-commit dry-build for web-arm (commit and let the installed hook run it; do not run the test script manually). - [ ] A PR is opened (never a direct push to main), Conventional-Commits style, and its description lists every secret the maintainer must add to the sops files (names + which file) and every external action required before deploy: Storage-Box sub-account + snapshots, DNS record for paperless.cloonar.com, `scan@cloonar.com` mailbox, Keycloak group membership. **Out of scope:** - Editing any `*.sops.yaml` secret values (name what's needed in the PR instead). - The mail-host mailbox, DNS record, and Storage-Box sub-account themselves (human actions). - Runtime UI configuration: IMAP account, mail rules, owner-assigning workflows, user permissions. - Office-document support (Tika/Gotenberg), any network consume share, Authelia forward-auth on the vhost.

dominik.polakovics added the

enhancement

ready-for-agent

labels

2026-08-23 12:40:22 +02:00

dominik.polakovics referenced this issue from a pull request that will close it,

2026-08-23 13:10:12 +02:00

feat(web-arm): install paperless-ngx #386