diff --git a/BACKUP_PROCEDURES.md b/BACKUP_PROCEDURES.md deleted file mode 100644 index 52106ca..0000000 --- a/BACKUP_PROCEDURES.md +++ /dev/null @@ -1,184 +0,0 @@ -# DocFast Backup & Disaster Recovery Procedures - -## Overview -DocFast now uses BorgBackup for full disaster recovery backups. The system backs up all critical components needed to restore the service on a new server. - -## What is Backed Up -- **PostgreSQL database** - Full database dump with schema and data -- **Docker volumes** - Application data and files -- **Nginx configuration** - Web server configuration -- **SSL certificates** - Let's Encrypt certificates and keys -- **Crontabs** - Scheduled tasks -- **OpenDKIM keys** - Email authentication keys -- **DocFast application files** - docker-compose.yml, .env, scripts -- **System information** - Installed packages, enabled services, disk usage - -## Backup Location & Schedule - -### Current Setup (Local) -- **Location**: `/opt/borg-backups/docfast` -- **Schedule**: Daily at 03:00 UTC -- **Retention**: 7 daily + 4 weekly + 3 monthly backups -- **Compression**: LZ4 (fast compression/decompression) -- **Encryption**: repokey mode (encrypted with passphrase) - -### Security -- **Passphrase**: `docfast-backup-YYYY` (where YYYY is current year) -- **Key backup**: Stored in `/opt/borg-backups/docfast-key-backup.txt` -- **⚠️ IMPORTANT**: Both passphrase AND key are required for restore! - -## Scripts - -### Backup Script: `/opt/docfast-borg-backup.sh` -- Automated backup creation -- Runs via cron daily at 03:00 UTC -- Logs to `/var/log/docfast-backup.log` -- Auto-prunes old backups - -### Restore Script: `/opt/docfast-borg-restore.sh` -- List available backups: `./docfast-borg-restore.sh list` -- Restore specific backup: `./docfast-borg-restore.sh restore docfast-YYYY-MM-DD_HHMM` -- Restore latest backup: `./docfast-borg-restore.sh restore latest` - -## Manual Backup Commands - -```bash -# Run backup manually -/opt/docfast-borg-backup.sh - -# List all backups -export BORG_PASSPHRASE="docfast-backup-$(date +%Y)" -borg list /opt/borg-backups/docfast - -# Show repository info -borg info /opt/borg-backups/docfast - -# Show specific backup contents -borg list /opt/borg-backups/docfast::docfast-2026-02-15_1103 -``` - -## Disaster Recovery Procedure - -### Complete Server Rebuild -If the entire server is lost, follow these steps on a new server: - -1. **Install dependencies**: - ```bash - apt update && apt install -y docker.io docker-compose postgresql-16 nginx borgbackup - systemctl enable postgresql docker - ``` - -2. **Copy backup data**: - - Transfer `/opt/borg-backups/` directory to new server - - Transfer `/opt/borg-backups/docfast-key-backup.txt` - -3. **Import Borg key**: - ```bash - export BORG_PASSPHRASE="docfast-backup-2026" - borg key import /opt/borg-backups/docfast /opt/borg-backups/docfast-key-backup.txt - ``` - -4. **Restore latest backup**: - ```bash - /opt/docfast-borg-restore.sh restore latest - ``` - -5. **Follow manual restore steps** (shown by restore script): - - Stop services - - Restore database - - Restore configuration files - - Set permissions - - Start services - -### Database-Only Recovery -If only the database needs restoration: - -```bash -# Stop DocFast -cd /opt/docfast && docker-compose down - -# Restore database -export BORG_PASSPHRASE="docfast-backup-$(date +%Y)" -cd /tmp -borg extract /opt/borg-backups/docfast::docfast-YYYY-MM-DD_HHMM -sudo -u postgres dropdb docfast -sudo -u postgres createdb -O docfast docfast -export PGPASSFILE="/root/.pgpass" -pg_restore -d docfast /tmp/tmp/docfast-backup-*/docfast-db.dump - -# Restart DocFast -cd /opt/docfast && docker-compose up -d -``` - -## Migration to Off-Site Storage - -### Option 1: Hetzner Storage Box (Recommended) -Manual setup required (Hetzner Storage Box API not available): - -1. **Purchase Hetzner Storage Box** - - Minimum 10GB size - - Enable SSH access in Hetzner Console - -2. **Configure SSH access**: - ```bash - # Generate SSH key for storage box - ssh-keygen -t ed25519 -f /root/.ssh/hetzner-storage-box - - # Add public key to storage box in Hetzner Console - cat /root/.ssh/hetzner-storage-box.pub - ``` - -3. **Update backup script**: - Change `BORG_REPO` in `/opt/docfast-borg-backup.sh`: - ```bash - BORG_REPO="ssh://uXXXXXX@uXXXXXX.your-storagebox.de:23/./docfast-backups" - ``` - -4. **Initialize remote repository**: - ```bash - export BORG_PASSPHRASE="docfast-backup-$(date +%Y)" - borg init --encryption=repokey ssh://uXXXXXX@uXXXXXX.your-storagebox.de:23/./docfast-backups - ``` - -### Option 2: AWS S3/Glacier -Use rclone + borg for S3 storage (requires investor approval for AWS costs). - -## Monitoring & Maintenance - -### Check Backup Status -```bash -# View recent backup logs -tail -f /var/log/docfast-backup.log - -# Check repository size and stats -export BORG_PASSPHRASE="docfast-backup-$(date +%Y)" -borg info /opt/borg-backups/docfast -``` - -### Manual Cleanup -```bash -# Prune old backups manually -borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 3 /opt/borg-backups/docfast - -# Compact repository -borg compact /opt/borg-backups/docfast -``` - -### Repository Health Check -```bash -# Check repository consistency -borg check --verify-data /opt/borg-backups/docfast -``` - -## Important Notes - -1. **Test restores regularly** - Run restore test monthly -2. **Monitor backup logs** - Check for failures in `/var/log/docfast-backup.log` -3. **Keep key safe** - Store `/opt/borg-backups/docfast-key-backup.txt` securely off-site -4. **Update passphrase annually** - Change to new year format when year changes -5. **Local storage limit** - Current server has ~19GB available, monitor usage - -## Migration Timeline -- **Immediate**: Local BorgBackup operational (✅ Complete) -- **Phase 2**: Off-site storage setup (requires Storage Box purchase or AWS approval) -- **Phase 3**: Automated off-site testing and monitoring \ No newline at end of file diff --git a/CI-CD-SETUP-COMPLETE.md b/CI-CD-SETUP-COMPLETE.md deleted file mode 100644 index d1aee96..0000000 --- a/CI-CD-SETUP-COMPLETE.md +++ /dev/null @@ -1,121 +0,0 @@ -# DocFast CI/CD Pipeline Setup - COMPLETED ✅ - -## What Was Implemented - -### ✅ Forgejo Actions Workflow -- **File**: `.forgejo/workflows/deploy.yml` -- **Trigger**: Push to `main` branch -- **Process**: - 1. SSH to production server (167.235.156.214) - 2. Pull latest code from git - 3. Tag current Docker image for rollback (`rollback-YYYYMMDD-HHMMSS`) - 4. Build new Docker image with `--no-cache` - 5. Stop current services (30s graceful timeout) - 6. Start new services with `docker compose up -d` - 7. Health check at `http://127.0.0.1:3100/health` (30 attempts, 5s intervals) - 8. **Auto-rollback** if health check fails - 9. Cleanup old rollback images (keeps last 5) - -### ✅ Rollback Mechanism -- **Automatic**: Built into the deployment workflow -- **Manual Script**: `scripts/rollback.sh` for emergency use -- **Image Tagging**: Previous images tagged with timestamps -- **Auto-cleanup**: Removes old rollback images automatically - -### ✅ Documentation -- **`DEPLOYMENT.md`**: Complete deployment guide -- **`CI-CD-SETUP-COMPLETE.md`**: This summary -- **Inline comments**: Detailed workflow documentation - -### ✅ Git Integration -- Repository: `git@git.cloonar.com:openclawd/docfast.git` -- SSH access configured with key: `/home/openclaw/.ssh/docfast` -- All CI/CD files committed and pushed successfully - -## What Needs Manual Setup (5 minutes) - -### 🔧 Repository Secrets -Go to: https://git.cloonar.com/openclawd/docfast/settings/actions/secrets - -Add these 3 secrets: -1. **SERVER_HOST**: `167.235.156.214` -2. **SERVER_USER**: `root` -3. **SSH_PRIVATE_KEY**: (copy content from `/home/openclaw/.ssh/docfast`) - -### 🧪 Test the Pipeline -1. Once secrets are added, push any change to main branch -2. Check Actions tab: https://git.cloonar.com/openclawd/docfast/actions -3. Watch deployment progress -4. Verify with: `curl http://127.0.0.1:3100/health` - -## How to Trigger Deployments - -- **Automatic**: Any push to `main` branch -- **Manual**: Push a trivial change (already prepared: VERSION file) - -## How to Rollback - -### Automatic Rollback -- Happens automatically if new deployment fails health checks -- No manual intervention required - -### Manual Rollback Options -```bash -# Option 1: Use the rollback script -ssh root@167.235.156.214 -cd /root/docfast -./scripts/rollback.sh - -# Option 2: Manual Docker commands -ssh root@167.235.156.214 -docker compose down -docker images | grep rollback # Find latest rollback image -docker tag docfast-docfast:rollback-YYYYMMDD-HHMMSS docfast-docfast:latest -docker compose up -d -``` - -## Monitoring Commands - -```bash -# Health check -curl http://127.0.0.1:3100/health - -# Service status -docker compose ps - -# View logs -docker compose logs -f docfast - -# Check rollback images available -docker images | grep docfast-docfast -``` - -## Files Added/Modified - -``` -.forgejo/workflows/deploy.yml # Main deployment workflow -scripts/rollback.sh # Emergency rollback script -scripts/setup-secrets.sh # Helper script (API had auth issues) -DEPLOYMENT.md # Deployment documentation -CI-CD-SETUP-COMPLETE.md # This summary -VERSION # Test file for pipeline testing -``` - -## Next Steps - -1. **Set up secrets** in Forgejo (5 minutes) -2. **Test deployment** by making a small change -3. **Verify** the health check endpoint works -4. **Document** any environment-specific adjustments needed - -## Success Criteria ✅ - -- [x] Forgejo Actions available and configured -- [x] Deployment workflow created and tested (syntax) -- [x] Rollback mechanism implemented (automatic + manual) -- [x] Health check integration (`/health` endpoint) -- [x] Git repository integration working -- [x] Documentation complete -- [x] Test change ready for pipeline verification - -**Ready for production use once secrets are configured!** 🚀 \ No newline at end of file diff --git a/Dockerfile.backup b/Dockerfile.backup deleted file mode 100644 index bdc953a..0000000 --- a/Dockerfile.backup +++ /dev/null @@ -1,19 +0,0 @@ -FROM node:22-bookworm-slim - -# Install Chromium (works on ARM and x86) -RUN apt-get update && apt-get install -y --no-install-recommends \ - chromium fonts-liberation \ - && rm -rf /var/lib/apt/lists/* - -ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true -ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium - -WORKDIR /app -COPY package*.json ./ -RUN npm ci --omit=dev -COPY dist/ dist/ -COPY public/ public/ - -ENV PORT=3100 -EXPOSE 3100 -CMD ["node", "dist/index.js"]