docfast/infrastructure
openclawd 76c1cc3bfb Add comprehensive backup and restore infrastructure
- scripts/borg-backup.sh: Complete disaster recovery backup (PostgreSQL, Docker volumes, configs, SSL certs, DKIM keys)
- scripts/borg-restore.sh: Automated restore from Borg backups with manual verification steps
- scripts/rollback.sh: Quick application rollback using Docker image tags
- Enhanced setup.sh with BorgBackup installation and cron job setup
- Updated README with detailed backup strategy documentation (2-tier: SQLite + Borg)

Backup retention: 7 daily + 4 weekly + 3 monthly Borg archives + 7 days SQLite snapshots
2026-02-15 11:06:42 +00:00
..
nginx Add complete infrastructure automation and documentation 2026-02-15 11:04:34 +00:00
postfix Add complete infrastructure automation and documentation 2026-02-15 11:04:34 +00:00
.env.template Add complete infrastructure automation and documentation 2026-02-15 11:04:34 +00:00
docker-compose.yml Add complete infrastructure automation and documentation 2026-02-15 11:04:34 +00:00
README.md Add comprehensive backup and restore infrastructure 2026-02-15 11:06:42 +00:00
setup.sh Add comprehensive backup and restore infrastructure 2026-02-15 11:06:42 +00:00

DocFast Infrastructure Guide

Complete disaster recovery and deployment guide for DocFast.

Quick Start (New Server Deployment)

1. Prerequisites

  • Fresh Ubuntu 24.04 LTS server
  • Root access
  • Domain name pointing to server IP
  • Stripe account with webhook configured

2. Automated Setup

# Clone the repository
git clone ssh://forgejo@git.cloonar.com/openclawd/docfast.git
cd docfast/infrastructure

# Run the setup script as root
chmod +x setup.sh
./setup.sh

# Follow the post-setup instructions

3. Manual Configuration Required

After running setup.sh, complete these manual steps:

SSL Certificate

certbot --nginx -d docfast.dev -d www.docfast.dev

DKIM DNS Record

Add this TXT record to your DNS:

mail._domainkey.docfast.dev

Get the value from: /etc/opendkim/keys/docfast.dev/mail.txt

Environment Variables

cd /opt/docfast
cp infrastructure/.env.template .env
# Edit .env with real values

Start the Application

cd /opt/docfast
cp infrastructure/docker-compose.yml .
docker-compose up -d

Complete Manual Setup (Step by Step)

If the automated script fails or you prefer manual setup:

System Packages

apt update && apt upgrade -y
apt install -y nginx postfix opendkim opendkim-tools certbot \
    python3-certbot-nginx ufw docker.io docker-compose-plugin \
    git sqlite3 postgresql postgresql-contrib

Firewall Configuration

ufw --force enable
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow from 172.16.0.0/12 to any port 25 comment "Docker SMTP relay"
ufw allow from 172.16.0.0/12 to any port 5432 comment "Docker PostgreSQL"

PostgreSQL Setup

sudo -u postgres createuser -D -A -P docfast
sudo -u postgres createdb -O docfast docfast

# Edit /etc/postgresql/16/main/postgresql.conf
echo "listen_addresses = '*'" >> /etc/postgresql/16/main/postgresql.conf

# Edit /etc/postgresql/16/main/pg_hba.conf
echo "host docfast docfast 172.17.0.0/16 md5" >> /etc/postgresql/16/main/pg_hba.conf
echo "host docfast docfast 172.18.0.0/16 md5" >> /etc/postgresql/16/main/pg_hba.conf

systemctl restart postgresql

Nginx Configuration

cp nginx/docfast.dev /etc/nginx/sites-available/
ln -s /etc/nginx/sites-available/docfast.dev /etc/nginx/sites-enabled/
rm /etc/nginx/sites-enabled/default
nginx -t
systemctl reload nginx

Postfix & OpenDKIM

cp postfix/main.cf /etc/postfix/
cp postfix/opendkim.conf /etc/opendkim.conf
cp postfix/TrustedHosts /etc/opendkim/

# Generate DKIM keys
mkdir -p /etc/opendkim/keys/docfast.dev
cd /etc/opendkim/keys/docfast.dev
opendkim-genkey -s mail -d docfast.dev
chown opendkim:opendkim mail.private mail.txt
chmod 600 mail.private

systemctl restart postfix opendkim

Application Deployment

useradd -r -m -s /bin/bash docfast
usermod -aG docker docfast
mkdir -p /opt/docfast
chown docfast:docfast /opt/docfast

cd /opt/docfast
# Copy your source code here
cp infrastructure/docker-compose.yml .
cp infrastructure/.env.template .env
# Edit .env with real values

docker-compose up -d

Backup System

# Install BorgBackup
apt install -y borgbackup

# Create backup directories
mkdir -p /opt/docfast-backups /opt/borg-backups

# Copy backup scripts
cp scripts/docfast-backup.sh /opt/
cp scripts/borg-backup.sh /opt/
cp scripts/borg-restore.sh /opt/
cp scripts/rollback.sh /opt/
chmod +x /opt/docfast-backup.sh /opt/borg-backup.sh /opt/borg-restore.sh /opt/rollback.sh

# Add to root crontab
echo "0 */6 * * * /opt/docfast-backup.sh >> /var/log/docfast-backup.log 2>&1" | crontab -
echo "0 3 * * * /opt/borg-backup.sh >> /var/log/borg-backup.log 2>&1" | crontab -

Backup Strategy

DocFast uses a two-tier backup strategy for comprehensive data protection:

1. SQLite Database Backups (Every 6 hours)

  • Script: /opt/docfast-backup.sh
  • Frequency: Every 6 hours via cron
  • Retention: 7 days of backups (28 files), plus 4 weekly copies
  • Storage: /opt/docfast-backups/
  • Method: SQLite .backup command with integrity verification

2. Complete System Backups (Daily)

  • Script: /opt/borg-backup.sh
  • Frequency: Daily at 03:00 UTC via cron
  • Retention: 7 daily + 4 weekly + 3 monthly
  • Storage: /opt/borg-backups/docfast
  • Includes:
    • PostgreSQL database dump
    • Docker volumes (complete application data)
    • Nginx configuration
    • SSL certificates (Let's Encrypt)
    • OpenDKIM keys and configuration
    • Cron jobs and system configurations
    • Application files (.env, docker-compose.yml)
    • System information (packages, services)

Backup Management Commands

# List available Borg backups
/opt/borg-restore.sh list

# Restore from latest backup (creates restore directory)
/opt/borg-restore.sh restore latest

# Restore from specific backup
/opt/borg-restore.sh restore docfast-2026-02-15_0300

# Quick rollback (Docker image only)
/opt/rollback.sh

Disaster Recovery Procedures

Complete Server Failure

  1. Provision new server with same OS version
  2. Run setup script from this repository
  3. Restore DNS records to point to new server
  4. Copy backups from off-site storage to /opt/docfast-backups/
  5. Restore database:
    docker-compose down
    docker volume rm docfast_docfast-data
    docker volume create docfast_docfast-data
    cp /opt/docfast-backups/docfast-weekly-LATEST.db \
       /var/lib/docker/volumes/docfast_docfast-data/_data/docfast.db
    docker-compose up -d
    
  6. Verify SSL certificates with certbot certificates
  7. Test email delivery and DKIM signing

Database Corruption

cd /opt/docfast
docker-compose down

# Find latest good backup
ls -la /opt/docfast-backups/

# Restore from backup
cp /opt/docfast-backups/docfast-daily-LATEST.db \
   /var/lib/docker/volumes/docfast_docfast-data/_data/docfast.db

docker-compose up -d

Email Delivery Issues

Check DKIM setup:

# Verify DKIM key is readable
sudo -u opendkim cat /etc/opendkim/keys/docfast.dev/mail.private

# Check OpenDKIM is signing
tail -f /var/log/mail.log

# Test email sending
echo "Test email" | mail -s "Test" test@example.com

SSL Certificate Issues

# Check certificate status
certbot certificates

# Renew if needed
certbot renew --dry-run
certbot renew

# Fix nginx config if needed
nginx -t
systemctl reload nginx

Monitoring & Maintenance

Daily Checks

  • Application health: curl https://docfast.dev/health
  • Docker containers: docker ps
  • Disk space: df -h
  • Backup status: ls -la /opt/docfast-backups/

Weekly Checks

  • SSL certificate expiry: certbot certificates
  • Email delivery test
  • System updates: apt list --upgradable
  • Log rotation: du -sh /var/log/

Monthly Tasks

  • Review backup retention
  • Update system packages
  • Review firewall rules: ufw status
  • Check for failed login attempts: grep "Failed password" /var/log/auth.log

Environment Variables Reference

Variable Required Description Example
STRIPE_SECRET_KEY Stripe API secret key sk_live_...
STRIPE_WEBHOOK_SECRET Stripe webhook endpoint secret whsec_...
BASE_URL Application base URL https://docfast.dev
API_KEYS Comma-separated API keys key1,key2,key3
PRO_KEYS Comma-separated pro API keys prokey1,prokey2
DATABASE_PASSWORD PostgreSQL password secure_password_123

DNS Records Required

Type Name Value TTL
A docfast.dev SERVER_IP 300
A www.docfast.dev SERVER_IP 300
TXT mail._domainkey.docfast.dev DKIM_PUBLIC_KEY 300
MX docfast.dev docfast.dev 300
TXT docfast.dev v=spf1 mx ~all 300

Stripe Configuration

Required webhook events:

  • checkout.session.completed
  • invoice.payment_succeeded
  • customer.subscription.created
  • customer.subscription.updated
  • customer.subscription.deleted

Webhook URL: https://docfast.dev/api/stripe/webhook

Security Considerations

  • Server runs on non-standard SSH port (change from 22)
  • Fail2ban recommended for brute force protection
  • Regular security updates via unattended-upgrades
  • Database backups encrypted at rest
  • API keys rotated regularly
  • Monitor application logs for suspicious activity

Troubleshooting

Common Issues

Container won't start: Check logs with docker-compose logs -f

Database connection errors: Verify PostgreSQL is running and Docker networks are configured

Email not sending: Check postfix logs: tail -f /var/log/mail.log

SSL certificate errors: Verify domain DNS and run certbot --nginx

High memory usage: Monitor with docker stats and adjust container limits

Log Locations

  • Application: docker-compose logs
  • Nginx: /var/log/nginx/
  • Postfix: /var/log/mail.log
  • System: /var/log/syslog
  • Backups: /var/log/docfast-backup.log