Add BorgBackup disaster recovery system
- Full backup of PostgreSQL, Docker volumes, nginx config, SSL certs, crontabs, OpenDKIM - Daily backups at 03:00 UTC with 7d/4w/3m retention - Local storage at /opt/borg-backups/docfast - Restore testing verified - Documentation for disaster recovery procedures
This commit is contained in:
OpenClaw
parent
3820d7ea4d
commit
ef84279eae
3 changed files with 496 additions and 0 deletions
184
BACKUP_PROCEDURES.md
Normal file
184
BACKUP_PROCEDURES.md
Normal file
|
|
@ -0,0 +1,184 @@
|
|||
# DocFast Backup & Disaster Recovery Procedures
|
||||
|
||||
## Overview
|
||||
DocFast now uses BorgBackup for full disaster recovery backups. The system backs up all critical components needed to restore the service on a new server.
|
||||
|
||||
## What is Backed Up
|
||||
- **PostgreSQL database** - Full database dump with schema and data
|
||||
- **Docker volumes** - Application data and files
|
||||
- **Nginx configuration** - Web server configuration
|
||||
- **SSL certificates** - Let's Encrypt certificates and keys
|
||||
- **Crontabs** - Scheduled tasks
|
||||
- **OpenDKIM keys** - Email authentication keys
|
||||
- **DocFast application files** - docker-compose.yml, .env, scripts
|
||||
- **System information** - Installed packages, enabled services, disk usage
|
||||
|
||||
## Backup Location & Schedule
|
||||
|
||||
### Current Setup (Local)
|
||||
- **Location**: `/opt/borg-backups/docfast`
|
||||
- **Schedule**: Daily at 03:00 UTC
|
||||
- **Retention**: 7 daily + 4 weekly + 3 monthly backups
|
||||
- **Compression**: LZ4 (fast compression/decompression)
|
||||
- **Encryption**: repokey mode (encrypted with passphrase)
|
||||
|
||||
### Security
|
||||
- **Passphrase**: `docfast-backup-YYYY` (where YYYY is current year)
|
||||
- **Key backup**: Stored in `/opt/borg-backups/docfast-key-backup.txt`
|
||||
- **⚠️ IMPORTANT**: Both passphrase AND key are required for restore!
|
||||
|
||||
## Scripts
|
||||
|
||||
### Backup Script: `/opt/docfast-borg-backup.sh`
|
||||
- Automated backup creation
|
||||
- Runs via cron daily at 03:00 UTC
|
||||
- Logs to `/var/log/docfast-backup.log`
|
||||
- Auto-prunes old backups
|
||||
|
||||
### Restore Script: `/opt/docfast-borg-restore.sh`
|
||||
- List available backups: `./docfast-borg-restore.sh list`
|
||||
- Restore specific backup: `./docfast-borg-restore.sh restore docfast-YYYY-MM-DD_HHMM`
|
||||
- Restore latest backup: `./docfast-borg-restore.sh restore latest`
|
||||
|
||||
## Manual Backup Commands
|
||||
|
||||
```bash
|
||||
# Run backup manually
|
||||
/opt/docfast-borg-backup.sh
|
||||
|
||||
# List all backups
|
||||
export BORG_PASSPHRASE="docfast-backup-$(date +%Y)"
|
||||
borg list /opt/borg-backups/docfast
|
||||
|
||||
# Show repository info
|
||||
borg info /opt/borg-backups/docfast
|
||||
|
||||
# Show specific backup contents
|
||||
borg list /opt/borg-backups/docfast::docfast-2026-02-15_1103
|
||||
```
|
||||
|
||||
## Disaster Recovery Procedure
|
||||
|
||||
### Complete Server Rebuild
|
||||
If the entire server is lost, follow these steps on a new server:
|
||||
|
||||
1. **Install dependencies**:
|
||||
```bash
|
||||
apt update && apt install -y docker.io docker-compose postgresql-16 nginx borgbackup
|
||||
systemctl enable postgresql docker
|
||||
```
|
||||
|
||||
2. **Copy backup data**:
|
||||
- Transfer `/opt/borg-backups/` directory to new server
|
||||
- Transfer `/opt/borg-backups/docfast-key-backup.txt`
|
||||
|
||||
3. **Import Borg key**:
|
||||
```bash
|
||||
export BORG_PASSPHRASE="docfast-backup-2026"
|
||||
borg key import /opt/borg-backups/docfast /opt/borg-backups/docfast-key-backup.txt
|
||||
```
|
||||
|
||||
4. **Restore latest backup**:
|
||||
```bash
|
||||
/opt/docfast-borg-restore.sh restore latest
|
||||
```
|
||||
|
||||
5. **Follow manual restore steps** (shown by restore script):
|
||||
- Stop services
|
||||
- Restore database
|
||||
- Restore configuration files
|
||||
- Set permissions
|
||||
- Start services
|
||||
|
||||
### Database-Only Recovery
|
||||
If only the database needs restoration:
|
||||
|
||||
```bash
|
||||
# Stop DocFast
|
||||
cd /opt/docfast && docker-compose down
|
||||
|
||||
# Restore database
|
||||
export BORG_PASSPHRASE="docfast-backup-$(date +%Y)"
|
||||
cd /tmp
|
||||
borg extract /opt/borg-backups/docfast::docfast-YYYY-MM-DD_HHMM
|
||||
sudo -u postgres dropdb docfast
|
||||
sudo -u postgres createdb -O docfast docfast
|
||||
export PGPASSFILE="/root/.pgpass"
|
||||
pg_restore -d docfast /tmp/tmp/docfast-backup-*/docfast-db.dump
|
||||
|
||||
# Restart DocFast
|
||||
cd /opt/docfast && docker-compose up -d
|
||||
```
|
||||
|
||||
## Migration to Off-Site Storage
|
||||
|
||||
### Option 1: Hetzner Storage Box (Recommended)
|
||||
Manual setup required (Hetzner Storage Box API not available):
|
||||
|
||||
1. **Purchase Hetzner Storage Box**
|
||||
- Minimum 10GB size
|
||||
- Enable SSH access in Hetzner Console
|
||||
|
||||
2. **Configure SSH access**:
|
||||
```bash
|
||||
# Generate SSH key for storage box
|
||||
ssh-keygen -t ed25519 -f /root/.ssh/hetzner-storage-box
|
||||
|
||||
# Add public key to storage box in Hetzner Console
|
||||
cat /root/.ssh/hetzner-storage-box.pub
|
||||
```
|
||||
|
||||
3. **Update backup script**:
|
||||
Change `BORG_REPO` in `/opt/docfast-borg-backup.sh`:
|
||||
```bash
|
||||
BORG_REPO="ssh://uXXXXXX@uXXXXXX.your-storagebox.de:23/./docfast-backups"
|
||||
```
|
||||
|
||||
4. **Initialize remote repository**:
|
||||
```bash
|
||||
export BORG_PASSPHRASE="docfast-backup-$(date +%Y)"
|
||||
borg init --encryption=repokey ssh://uXXXXXX@uXXXXXX.your-storagebox.de:23/./docfast-backups
|
||||
```
|
||||
|
||||
### Option 2: AWS S3/Glacier
|
||||
Use rclone + borg for S3 storage (requires investor approval for AWS costs).
|
||||
|
||||
## Monitoring & Maintenance
|
||||
|
||||
### Check Backup Status
|
||||
```bash
|
||||
# View recent backup logs
|
||||
tail -f /var/log/docfast-backup.log
|
||||
|
||||
# Check repository size and stats
|
||||
export BORG_PASSPHRASE="docfast-backup-$(date +%Y)"
|
||||
borg info /opt/borg-backups/docfast
|
||||
```
|
||||
|
||||
### Manual Cleanup
|
||||
```bash
|
||||
# Prune old backups manually
|
||||
borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 3 /opt/borg-backups/docfast
|
||||
|
||||
# Compact repository
|
||||
borg compact /opt/borg-backups/docfast
|
||||
```
|
||||
|
||||
### Repository Health Check
|
||||
```bash
|
||||
# Check repository consistency
|
||||
borg check --verify-data /opt/borg-backups/docfast
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
|
||||
1. **Test restores regularly** - Run restore test monthly
|
||||
2. **Monitor backup logs** - Check for failures in `/var/log/docfast-backup.log`
|
||||
3. **Keep key safe** - Store `/opt/borg-backups/docfast-key-backup.txt` securely off-site
|
||||
4. **Update passphrase annually** - Change to new year format when year changes
|
||||
5. **Local storage limit** - Current server has ~19GB available, monitor usage
|
||||
|
||||
## Migration Timeline
|
||||
- **Immediate**: Local BorgBackup operational (✅ Complete)
|
||||
- **Phase 2**: Off-site storage setup (requires Storage Box purchase or AWS approval)
|
||||
- **Phase 3**: Automated off-site testing and monitoring
|
||||
Loading…
Add table
Add a link
Reference in a new issue