docfast/infrastructure/README.md

# DocFast Infrastructure Guide

Complete disaster recovery and deployment guide for DocFast.

## Quick Start (New Server Deployment)

### 1. Prerequisites

- Fresh Ubuntu 24.04 LTS server
- Root access
- Domain name pointing to server IP
- Stripe account with webhook configured

### 2. Automated Setup

```bash
# Clone the repository
git clone ssh://forgejo@git.cloonar.com/openclawd/docfast.git
cd docfast/infrastructure

# Run the setup script as root
chmod +x setup.sh
./setup.sh

# Follow the post-setup instructions
```

### 3. Manual Configuration Required

After running `setup.sh`, complete these manual steps:

#### SSL Certificate
```bash
certbot --nginx -d docfast.dev -d www.docfast.dev
```

#### DKIM DNS Record
Add this TXT record to your DNS:
```
mail._domainkey.docfast.dev
```
Get the value from: `/etc/opendkim/keys/docfast.dev/mail.txt`

#### Environment Variables
```bash
cd /opt/docfast
cp infrastructure/.env.template .env
# Edit .env with real values
```

#### Start the Application
```bash
cd /opt/docfast
cp infrastructure/docker-compose.yml .
docker-compose up -d
```

## Complete Manual Setup (Step by Step)

If the automated script fails or you prefer manual setup:

### System Packages
```bash
apt update && apt upgrade -y
apt install -y nginx postfix opendkim opendkim-tools certbot \
    python3-certbot-nginx ufw docker.io docker-compose-plugin \
    git sqlite3 postgresql postgresql-contrib
```

### Firewall Configuration
```bash
ufw --force enable
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow from 172.16.0.0/12 to any port 25 comment "Docker SMTP relay"
ufw allow from 172.16.0.0/12 to any port 5432 comment "Docker PostgreSQL"
```

### PostgreSQL Setup
```bash
sudo -u postgres createuser -D -A -P docfast
sudo -u postgres createdb -O docfast docfast

# Edit /etc/postgresql/16/main/postgresql.conf
echo "listen_addresses = '*'" >> /etc/postgresql/16/main/postgresql.conf

# Edit /etc/postgresql/16/main/pg_hba.conf
echo "host docfast docfast 172.17.0.0/16 md5" >> /etc/postgresql/16/main/pg_hba.conf
echo "host docfast docfast 172.18.0.0/16 md5" >> /etc/postgresql/16/main/pg_hba.conf

systemctl restart postgresql
```

### Nginx Configuration
```bash
cp nginx/docfast.dev /etc/nginx/sites-available/
ln -s /etc/nginx/sites-available/docfast.dev /etc/nginx/sites-enabled/
rm /etc/nginx/sites-enabled/default
nginx -t
systemctl reload nginx
```

### Postfix & OpenDKIM
```bash
cp postfix/main.cf /etc/postfix/
cp postfix/opendkim.conf /etc/opendkim.conf
cp postfix/TrustedHosts /etc/opendkim/

# Generate DKIM keys
mkdir -p /etc/opendkim/keys/docfast.dev
cd /etc/opendkim/keys/docfast.dev
opendkim-genkey -s mail -d docfast.dev
chown opendkim:opendkim mail.private mail.txt
chmod 600 mail.private

systemctl restart postfix opendkim
```

### Application Deployment
```bash
useradd -r -m -s /bin/bash docfast
usermod -aG docker docfast
mkdir -p /opt/docfast
chown docfast:docfast /opt/docfast

cd /opt/docfast
# Copy your source code here
cp infrastructure/docker-compose.yml .
cp infrastructure/.env.template .env
# Edit .env with real values

docker-compose up -d
```

### Backup System
```bash
# Install BorgBackup
apt install -y borgbackup

# Create backup directories
mkdir -p /opt/docfast-backups /opt/borg-backups

# Copy backup scripts
cp scripts/docfast-backup.sh /opt/
cp scripts/borg-backup.sh /opt/
cp scripts/borg-restore.sh /opt/
cp scripts/rollback.sh /opt/
chmod +x /opt/docfast-backup.sh /opt/borg-backup.sh /opt/borg-restore.sh /opt/rollback.sh

# Add to root crontab
echo "0 */6 * * * /opt/docfast-backup.sh >> /var/log/docfast-backup.log 2>&1" | crontab -
echo "0 3 * * * /opt/borg-backup.sh >> /var/log/borg-backup.log 2>&1" | crontab -
```

## Backup Strategy

DocFast uses a two-tier backup strategy for comprehensive data protection:

### 1. SQLite Database Backups (Every 6 hours)
- **Script**: `/opt/docfast-backup.sh`
- **Frequency**: Every 6 hours via cron
- **Retention**: 7 days of backups (28 files), plus 4 weekly copies
- **Storage**: `/opt/docfast-backups/`
- **Method**: SQLite `.backup` command with integrity verification

### 2. Complete System Backups (Daily)
- **Script**: `/opt/borg-backup.sh` 
- **Frequency**: Daily at 03:00 UTC via cron
- **Retention**: 7 daily + 4 weekly + 3 monthly
- **Storage**: `/opt/borg-backups/docfast`
- **Includes**:
  - PostgreSQL database dump
  - Docker volumes (complete application data)
  - Nginx configuration
  - SSL certificates (Let's Encrypt)
  - OpenDKIM keys and configuration
  - Cron jobs and system configurations
  - Application files (.env, docker-compose.yml)
  - System information (packages, services)

### Backup Management Commands
```bash
# List available Borg backups
/opt/borg-restore.sh list

# Restore from latest backup (creates restore directory)
/opt/borg-restore.sh restore latest

# Restore from specific backup
/opt/borg-restore.sh restore docfast-2026-02-15_0300

# Quick rollback (Docker image only)
/opt/rollback.sh
```

## Disaster Recovery Procedures

### Complete Server Failure

1. **Provision new server** with same OS version
2. **Run setup script** from this repository
3. **Restore DNS** records to point to new server
4. **Copy backups** from off-site storage to `/opt/docfast-backups/`
5. **Restore database**:
   ```bash
   docker-compose down
   docker volume rm docfast_docfast-data
   docker volume create docfast_docfast-data
   cp /opt/docfast-backups/docfast-weekly-LATEST.db \
      /var/lib/docker/volumes/docfast_docfast-data/_data/docfast.db
   docker-compose up -d
   ```
6. **Verify SSL certificates** with `certbot certificates`
7. **Test email delivery** and DKIM signing

### Database Corruption

```bash
cd /opt/docfast
docker-compose down

# Find latest good backup
ls -la /opt/docfast-backups/

# Restore from backup
cp /opt/docfast-backups/docfast-daily-LATEST.db \
   /var/lib/docker/volumes/docfast_docfast-data/_data/docfast.db

docker-compose up -d
```

### Email Delivery Issues

Check DKIM setup:
```bash
# Verify DKIM key is readable
sudo -u opendkim cat /etc/opendkim/keys/docfast.dev/mail.private

# Check OpenDKIM is signing
tail -f /var/log/mail.log

# Test email sending
echo "Test email" | mail -s "Test" test@example.com
```

### SSL Certificate Issues

```bash
# Check certificate status
certbot certificates

# Renew if needed
certbot renew --dry-run
certbot renew

# Fix nginx config if needed
nginx -t
systemctl reload nginx
```

## Monitoring & Maintenance

### Daily Checks
- [ ] Application health: `curl https://docfast.dev/health`
- [ ] Docker containers: `docker ps`
- [ ] Disk space: `df -h`
- [ ] Backup status: `ls -la /opt/docfast-backups/`

### Weekly Checks
- [ ] SSL certificate expiry: `certbot certificates`
- [ ] Email delivery test
- [ ] System updates: `apt list --upgradable`
- [ ] Log rotation: `du -sh /var/log/`

### Monthly Tasks
- [ ] Review backup retention
- [ ] Update system packages
- [ ] Review firewall rules: `ufw status`
- [ ] Check for failed login attempts: `grep "Failed password" /var/log/auth.log`

## Environment Variables Reference

| Variable | Required | Description | Example |
|----------|----------|-------------|---------|
| `STRIPE_SECRET_KEY` | ✅ | Stripe API secret key | `sk_live_...` |
| `STRIPE_WEBHOOK_SECRET` | ✅ | Stripe webhook endpoint secret | `whsec_...` |
| `BASE_URL` | ✅ | Application base URL | `https://docfast.dev` |
| `API_KEYS` | ✅ | Comma-separated API keys | `key1,key2,key3` |
| `PRO_KEYS` | ✅ | Comma-separated pro API keys | `prokey1,prokey2` |
| `DATABASE_PASSWORD` | ✅ | PostgreSQL password | `secure_password_123` |

## DNS Records Required

| Type | Name | Value | TTL |
|------|------|-------|-----|
| A | docfast.dev | SERVER_IP | 300 |
| A | www.docfast.dev | SERVER_IP | 300 |
| TXT | mail._domainkey.docfast.dev | DKIM_PUBLIC_KEY | 300 |
| MX | docfast.dev | docfast.dev | 300 |
| TXT | docfast.dev | v=spf1 mx ~all | 300 |

## Stripe Configuration

Required webhook events:
- `checkout.session.completed`
- `invoice.payment_succeeded`
- `customer.subscription.created`
- `customer.subscription.updated`
- `customer.subscription.deleted`

Webhook URL: `https://docfast.dev/api/stripe/webhook`

## Security Considerations

- Server runs on non-standard SSH port (change from 22)
- Fail2ban recommended for brute force protection
- Regular security updates via unattended-upgrades
- Database backups encrypted at rest
- API keys rotated regularly
- Monitor application logs for suspicious activity

## Troubleshooting

### Common Issues

**Container won't start**: Check logs with `docker-compose logs -f`

**Database connection errors**: Verify PostgreSQL is running and Docker networks are configured

**Email not sending**: Check postfix logs: `tail -f /var/log/mail.log`

**SSL certificate errors**: Verify domain DNS and run `certbot --nginx`

**High memory usage**: Monitor with `docker stats` and adjust container limits

### Log Locations
- Application: `docker-compose logs`
- Nginx: `/var/log/nginx/`
- Postfix: `/var/log/mail.log`
- System: `/var/log/syslog`
- Backups: `/var/log/docfast-backup.log`