docfast/BACKUP_PROCEDURES.md

DocFast Backup & Disaster Recovery Procedures

Overview

DocFast now uses BorgBackup for full disaster recovery backups. The system backs up all critical components needed to restore the service on a new server.

What is Backed Up

• PostgreSQL database - Full database dump with schema and data
• Docker volumes - Application data and files
• Nginx configuration - Web server configuration
• SSL certificates - Let's Encrypt certificates and keys
• Crontabs - Scheduled tasks
• OpenDKIM keys - Email authentication keys
• DocFast application files - docker-compose.yml, .env, scripts
• System information - Installed packages, enabled services, disk usage

Backup Location & Schedule

Current Setup (Local)

• Location: /opt/borg-backups/docfast
• Schedule: Daily at 03:00 UTC
• Retention: 7 daily + 4 weekly + 3 monthly backups
• Compression: LZ4 (fast compression/decompression)
• Encryption: repokey mode (encrypted with passphrase)

Security

• Passphrase: docfast-backup-YYYY (where YYYY is current year)
• Key backup: Stored in /opt/borg-backups/docfast-key-backup.txt
• ⚠️ IMPORTANT: Both passphrase AND key are required for restore!

Scripts

Backup Script: /opt/docfast-borg-backup.sh

• Automated backup creation
• Runs via cron daily at 03:00 UTC
• Logs to /var/log/docfast-backup.log
• Auto-prunes old backups

Restore Script: /opt/docfast-borg-restore.sh

• List available backups: ./docfast-borg-restore.sh list
• Restore specific backup: ./docfast-borg-restore.sh restore docfast-YYYY-MM-DD_HHMM
• Restore latest backup: ./docfast-borg-restore.sh restore latest

Manual Backup Commands

# Run backup manually
/opt/docfast-borg-backup.sh

# List all backups
export BORG_PASSPHRASE="docfast-backup-$(date +%Y)"
borg list /opt/borg-backups/docfast

# Show repository info
borg info /opt/borg-backups/docfast

# Show specific backup contents
borg list /opt/borg-backups/docfast::docfast-2026-02-15_1103

Disaster Recovery Procedure

Complete Server Rebuild

If the entire server is lost, follow these steps on a new server:

1. Install dependencies:

   apt update && apt install -y docker.io docker-compose postgresql-16 nginx borgbackup
   systemctl enable postgresql docker
   

2. Copy backup data:

   • Transfer /opt/borg-backups/ directory to new server
   • Transfer /opt/borg-backups/docfast-key-backup.txt

3. Import Borg key:

   export BORG_PASSPHRASE="docfast-backup-2026"
   borg key import /opt/borg-backups/docfast /opt/borg-backups/docfast-key-backup.txt
   

4. Restore latest backup:

   /opt/docfast-borg-restore.sh restore latest
   

5. Follow manual restore steps (shown by restore script):

   • Stop services
   • Restore database
   • Restore configuration files
   • Set permissions
   • Start services

Database-Only Recovery

If only the database needs restoration:

# Stop DocFast
cd /opt/docfast && docker-compose down

# Restore database
export BORG_PASSPHRASE="docfast-backup-$(date +%Y)"
cd /tmp
borg extract /opt/borg-backups/docfast::docfast-YYYY-MM-DD_HHMM
sudo -u postgres dropdb docfast
sudo -u postgres createdb -O docfast docfast
export PGPASSFILE="/root/.pgpass"
pg_restore -d docfast /tmp/tmp/docfast-backup-*/docfast-db.dump

# Restart DocFast
cd /opt/docfast && docker-compose up -d

Migration to Off-Site Storage

Option 1: Hetzner Storage Box (Recommended)

Manual setup required (Hetzner Storage Box API not available):

1. Purchase Hetzner Storage Box

   • Minimum 10GB size
   • Enable SSH access in Hetzner Console

2. Configure SSH access:

   # Generate SSH key for storage box
   ssh-keygen -t ed25519 -f /root/.ssh/hetzner-storage-box
   
   # Add public key to storage box in Hetzner Console
   cat /root/.ssh/hetzner-storage-box.pub
   

3. Update backup script: Change BORG_REPO in /opt/docfast-borg-backup.sh:

   BORG_REPO="ssh://uXXXXXX@uXXXXXX.your-storagebox.de:23/./docfast-backups"
   

4. Initialize remote repository:

   export BORG_PASSPHRASE="docfast-backup-$(date +%Y)"
   borg init --encryption=repokey ssh://uXXXXXX@uXXXXXX.your-storagebox.de:23/./docfast-backups
   

Option 2: AWS S3/Glacier

Use rclone + borg for S3 storage (requires investor approval for AWS costs).

Monitoring & Maintenance

Check Backup Status

# View recent backup logs
tail -f /var/log/docfast-backup.log

# Check repository size and stats
export BORG_PASSPHRASE="docfast-backup-$(date +%Y)"
borg info /opt/borg-backups/docfast

Manual Cleanup

# Prune old backups manually
borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 3 /opt/borg-backups/docfast

# Compact repository
borg compact /opt/borg-backups/docfast

Repository Health Check

# Check repository consistency
borg check --verify-data /opt/borg-backups/docfast

Important Notes

1. Test restores regularly - Run restore test monthly
2. Monitor backup logs - Check for failures in /var/log/docfast-backup.log
3. Keep key safe - Store /opt/borg-backups/docfast-key-backup.txt securely off-site
4. Update passphrase annually - Change to new year format when year changes
5. Local storage limit - Current server has ~19GB available, monitor usage

Migration Timeline

• Immediate: Local BorgBackup operational (✅ Complete)
• Phase 2: Off-site storage setup (requires Storage Box purchase or AWS approval)
• Phase 3: Automated off-site testing and monitoring