feat: add fueltide backup
This commit is contained in:
parent
6d25a6074b
commit
bef415b591
4 changed files with 255 additions and 60 deletions
|
|
@ -42,6 +42,7 @@
|
||||||
./modules/scana11y.nix
|
./modules/scana11y.nix
|
||||||
|
|
||||||
./modules/wireguard.nix
|
./modules/wireguard.nix
|
||||||
|
./modules/fueltide-backup
|
||||||
];
|
];
|
||||||
|
|
||||||
nixpkgs.overlays = [
|
nixpkgs.overlays = [
|
||||||
|
|
|
||||||
129
hosts/web-arm/modules/fueltide-backup/RESTORATION.md
Normal file
129
hosts/web-arm/modules/fueltide-backup/RESTORATION.md
Normal file
|
|
@ -0,0 +1,129 @@
|
||||||
|
# Fueltide Supabase Restoration Runbook
|
||||||
|
|
||||||
|
Use this when the upstream Supabase project at `majxbigjafpzayzboxsf.supabase.co` is gone, broken, or you want to move to a new project.
|
||||||
|
|
||||||
|
## What this backup covers
|
||||||
|
|
||||||
|
The nightly `fueltide-backup.service` on `web-arm` produces three SQL files per run under `/var/backup/fueltide-supabase/<timestamp>/`:
|
||||||
|
|
||||||
|
- `roles.sql` — cluster roles (via `pg_dumpall --roles-only --no-role-passwords`)
|
||||||
|
- `schema.sql` — DDL: tables, functions, triggers, RLS policies, views, extensions, types (via `pg_dump --schema-only`)
|
||||||
|
- `data.sql` — all row data, including `auth.users`, `auth.identities`, `storage.objects` metadata (via `pg_dump --data-only`)
|
||||||
|
- `sha256.txt` — checksums for verification
|
||||||
|
|
||||||
|
These files are included in the nightly borgbackup run (03:00 UTC) and shipped to the Hetzner Storage Box at `u149513-sub8`.
|
||||||
|
|
||||||
|
## What this backup does **not** cover
|
||||||
|
|
||||||
|
- **Supabase Edge Functions** — lives in the `fueltide` app repo, deployed via `supabase functions deploy`. No action needed beyond redeploying from source.
|
||||||
|
- **Storage bucket files** — not in use for this project (only DB-backed data).
|
||||||
|
- **Control-plane settings** — auth providers, SMTP, email templates, API keys. These live in Supabase's dashboard, not the database. Must be reapplied manually (steps below).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Restoration steps
|
||||||
|
|
||||||
|
### 1. Provision a fresh Supabase project
|
||||||
|
|
||||||
|
Dashboard → New project. Use the same region (`eu-west-1`). Record:
|
||||||
|
- New **project ref** (20-char subdomain)
|
||||||
|
- New **database password**
|
||||||
|
- New **session pooler hostname** (Project Settings → Database → Connection string → Session pooler) — the cluster prefix (`aws-1-`, `aws-0-`, etc.) may differ from the old project.
|
||||||
|
|
||||||
|
### 2. Fetch the latest dump from borg
|
||||||
|
|
||||||
|
From `web-arm.cloonar.com`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
borg-list # find newest archive, e.g. web-arm-2026-04-24
|
||||||
|
mkdir -p /mnt/borg
|
||||||
|
borg-mount web-arm-2026-04-24 /mnt/borg
|
||||||
|
ls /mnt/borg/var/backup/fueltide-supabase/ # pick newest timestamped directory
|
||||||
|
cp -r /mnt/borg/var/backup/fueltide-supabase/<ts> /tmp/restore
|
||||||
|
borg umount /mnt/borg
|
||||||
|
|
||||||
|
cd /tmp/restore
|
||||||
|
sha256sum -c sha256.txt # verify integrity
|
||||||
|
```
|
||||||
|
|
||||||
|
If `web-arm` itself is lost, fetch from any machine with the borg SSH key + passphrase (secrets are in sops under `borg-ssh-key` / `borg-passphrase`).
|
||||||
|
|
||||||
|
### 3. Restore the database
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export NEW_URL="postgres://postgres.<new-ref>:<new-pw>@<new-pooler-host>:5432/postgres"
|
||||||
|
|
||||||
|
# roles (some will error because Supabase-managed roles already exist — safe to ignore)
|
||||||
|
psql "$NEW_URL" -f /tmp/restore/roles.sql || true
|
||||||
|
|
||||||
|
# schema
|
||||||
|
psql "$NEW_URL" -f /tmp/restore/schema.sql
|
||||||
|
|
||||||
|
# data
|
||||||
|
psql "$NEW_URL" -f /tmp/restore/data.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected noise that is safe to ignore:
|
||||||
|
- `role "supabase_admin" already exists`, same for `authenticator`, `service_role`, `anon`, `authenticated`, `dashboard_user`
|
||||||
|
- `extension "pg_graphql" already exists` (if schema uses `CREATE EXTENSION` without `IF NOT EXISTS` for any extension not pre-installed — rare)
|
||||||
|
- `schema "auth" already exists`
|
||||||
|
|
||||||
|
Stop and investigate if you see errors like `permission denied`, `syntax error`, or `duplicate key value`.
|
||||||
|
|
||||||
|
### 4. Redeploy Edge Functions from the app repo
|
||||||
|
|
||||||
|
From a checkout of the fueltide app repo:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
supabase link --project-ref <new-ref>
|
||||||
|
supabase functions deploy # deploys all functions in supabase/functions/
|
||||||
|
```
|
||||||
|
|
||||||
|
If specific function secrets are configured (via `supabase secrets set`), re-set them from the app repo's documented env values.
|
||||||
|
|
||||||
|
### 5. Reapply dashboard-only settings
|
||||||
|
|
||||||
|
These live in Supabase's control plane and are **not** in any dump:
|
||||||
|
|
||||||
|
| Setting | Location | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| Google OAuth provider | Authentication → Providers → Google | Client ID + secret from SOPS (commit `67e81d3` added these) |
|
||||||
|
| Apple OAuth provider | Authentication → Providers → Apple | Services ID + Team ID + Key ID + P8 key from SOPS |
|
||||||
|
| SMTP settings | Authentication → SMTP Settings | Sender `noreply@fueltide.io`, use the mail host's SMTP creds |
|
||||||
|
| Email templates | Authentication → Email Templates | Fueltide-branded magic link, confirm, recovery — bodies in commit `67e81d3` |
|
||||||
|
| API keys | Project Settings → API | A **new** `anon` and `service_role` are generated per project — copy them |
|
||||||
|
|
||||||
|
### 6. Update app clients
|
||||||
|
|
||||||
|
Update the iOS app (and any server-side callers) with:
|
||||||
|
|
||||||
|
- `SUPABASE_URL = https://<new-ref>.supabase.co`
|
||||||
|
- `SUPABASE_ANON_KEY = <new anon key>`
|
||||||
|
- `SUPABASE_SERVICE_ROLE_KEY = <new service role key>` (server-side only)
|
||||||
|
|
||||||
|
Update CSP in `hosts/web-arm/sites/fueltide.io.nix` (currently commented out, references `*.supabase.co`) if you reinstate it.
|
||||||
|
|
||||||
|
### 7. Smoke test
|
||||||
|
|
||||||
|
- Sign up + sign in via email magic link (confirms SMTP + email templates)
|
||||||
|
- Sign in via Google (confirms OAuth provider)
|
||||||
|
- Sign in via Apple (confirms OAuth provider)
|
||||||
|
- Read a known row from the largest app table (confirms data restored, RLS intact)
|
||||||
|
- Insert + read back a new row (confirms writes work)
|
||||||
|
- Call an edge function (confirms functions redeployed)
|
||||||
|
|
||||||
|
### 8. Update this backup service to point at the new project
|
||||||
|
|
||||||
|
Edit `hosts/web-arm/modules/fueltide-backup/default.nix`:
|
||||||
|
|
||||||
|
- Set `project = "<new-ref>"`
|
||||||
|
- Set `poolerHost = "<new-pooler-host>"` (the region + cluster may differ)
|
||||||
|
- If the new project is on a different Postgres major version, update `pg = pkgs.postgresql_XX`
|
||||||
|
|
||||||
|
Rotate the `fueltide-supabase-db-password` secret in `hosts/web-arm/secrets.yaml` via:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix-shell -p sops --run 'sops hosts/web-arm/secrets.yaml'
|
||||||
|
```
|
||||||
|
|
||||||
|
Deploy, then run `systemctl start fueltide-backup.service` manually on `web-arm` and verify a new dump lands under `/var/backup/fueltide-supabase/`.
|
||||||
64
hosts/web-arm/modules/fueltide-backup/default.nix
Normal file
64
hosts/web-arm/modules/fueltide-backup/default.nix
Normal file
|
|
@ -0,0 +1,64 @@
|
||||||
|
{ config, pkgs, ... }:
|
||||||
|
|
||||||
|
let
|
||||||
|
project = "majxbigjafpzayzboxsf";
|
||||||
|
poolerHost = "aws-1-eu-west-1.pooler.supabase.com";
|
||||||
|
outDir = "/var/backup/fueltide-supabase";
|
||||||
|
# retain local dumps for this many days; borg handles offsite retention
|
||||||
|
retainDays = 1;
|
||||||
|
# match the upstream Supabase Postgres major version
|
||||||
|
pg = pkgs.postgresql_17;
|
||||||
|
in {
|
||||||
|
sops.secrets.fueltide-supabase-db-password = { };
|
||||||
|
|
||||||
|
systemd.tmpfiles.rules = [ "d ${outDir} 0700 root root -" ];
|
||||||
|
|
||||||
|
systemd.services.fueltide-backup = {
|
||||||
|
description = "Dump upstream Supabase database for ${project}";
|
||||||
|
path = [ pg pkgs.coreutils pkgs.findutils ];
|
||||||
|
serviceConfig = {
|
||||||
|
Type = "oneshot";
|
||||||
|
User = "root";
|
||||||
|
LoadCredential = "db-password:${config.sops.secrets.fueltide-supabase-db-password.path}";
|
||||||
|
};
|
||||||
|
script = ''
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
export PGPASSWORD
|
||||||
|
PGPASSWORD=$(cat "$CREDENTIALS_DIRECTORY/db-password")
|
||||||
|
export PGHOST="${poolerHost}"
|
||||||
|
export PGPORT=5432
|
||||||
|
export PGUSER="postgres.${project}"
|
||||||
|
export PGDATABASE=postgres
|
||||||
|
|
||||||
|
TS=$(date -u +%Y%m%dT%H%M%SZ)
|
||||||
|
OUT="${outDir}/$TS"
|
||||||
|
mkdir -p "$OUT"
|
||||||
|
chmod 700 "$OUT"
|
||||||
|
|
||||||
|
# cluster roles (Supabase-managed roles already exist on a fresh project;
|
||||||
|
# restore errors for those are expected and benign)
|
||||||
|
pg_dumpall --roles-only --no-role-passwords > "$OUT/roles.sql"
|
||||||
|
|
||||||
|
# schema: tables, functions, triggers, RLS policies, views, extensions
|
||||||
|
pg_dump --schema-only --no-owner --no-privileges > "$OUT/schema.sql"
|
||||||
|
|
||||||
|
# data: all rows (includes auth.users, storage.objects metadata, etc.)
|
||||||
|
pg_dump --data-only --no-owner > "$OUT/data.sql"
|
||||||
|
|
||||||
|
( cd "$OUT" && sha256sum *.sql > sha256.txt )
|
||||||
|
|
||||||
|
find "${outDir}" -mindepth 1 -maxdepth 1 -type d \
|
||||||
|
-mtime +${toString retainDays} -exec rm -rf {} +
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
systemd.timers.fueltide-backup = {
|
||||||
|
wantedBy = [ "timers.target" ];
|
||||||
|
timerConfig = {
|
||||||
|
OnCalendar = "*-*-* 02:30:00";
|
||||||
|
Persistent = true;
|
||||||
|
RandomizedDelaySec = "10m";
|
||||||
|
};
|
||||||
|
};
|
||||||
|
}
|
||||||
File diff suppressed because one or more lines are too long
Loading…
Add table
Add a link
Reference in a new issue