diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..e525190 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,165 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +ParaClub AI Mailer is an automated email response system written in Go that: +- Fetches emails via IMAP from a configured mailbox +- Gathers context from configured URLs (HTML content extraction) +- Uses OpenRouter AI API to generate context-aware email replies +- Saves AI-generated responses as email drafts +- Processes emails sequentially with configurable polling intervals + +The system is designed as a long-running service with graceful shutdown handling. + +## Build and Run Commands + +```bash +# Install dependencies +go mod download + +# Build the application +go build -o paraclub-ai-mailer ./cmd/paraclub-ai-mailer + +# Run with default config (config.yaml in current directory) +./paraclub-ai-mailer + +# Run with custom config path +./paraclub-ai-mailer -config /path/to/config.yaml + +# Run tests (if any exist) +go test ./... + +# Format code +go fmt ./... + +# Vet code for issues +go vet ./... +``` + +## Configuration + +The application requires a `config.yaml` file. Use `config.yaml.example` as a template. + +Key configuration sections: +- **imap**: Server details, credentials, mailbox folders +- **ai**: OpenRouter API key, model selection, temperature, max tokens +- **context**: List of URLs to fetch as context for AI +- **polling**: Email check interval (e.g., "5m") +- **logging**: Log level and file path + +Credentials support three formats: +- Direct value: `password: "mypassword"` +- Environment variable: `password: "${IMAP_PASSWORD}"` +- File reference: `password: "file:///path/to/password.txt"` + +## Code Architecture + +### Module Structure + +``` +cmd/paraclub-ai-mailer/main.go # Entry point, orchestration +config/config.go # Configuration loading and validation +internal/ + ├── logger/logger.go # Centralized logging wrapper (logrus) + ├── imap/imap.go # IMAP client, email operations + ├── fetcher/fetcher.go # HTML content fetching and text extraction + └── ai/ai.go # OpenRouter AI integration +``` + +### Key Design Patterns + +**Email Processing Flow** (main.go:92-172): +1. Fetch unprocessed emails (UNSEEN flag) +2. Fetch context from all configured URLs +3. For each email: + - Detect language via AI + - Generate AI reply with context + - Save as draft with original email quoted + - Mark as processed (move to processed_box, mark as read) + +**IMAP Client** (internal/imap/imap.go): +- Connection handling with automatic reconnection (ensureConnection, reconnect) +- Folder path normalization with delimiter detection +- Gmail-specific logic in ensureFolder (uses "/" delimiter, handles system folders) +- Email content extraction handles multipart messages and quoted-printable encoding +- Draft formatting includes HTML response + quoted original message +- Panic recovery in SaveDraft and MarkAsProcessed + +**AI Integration** (internal/ai/ai.go): +- Two-stage process: language detection, then reply generation +- Language-aware responses (system prompt includes detected language) +- Output must be raw HTML (not markdown-wrapped) +- Retry logic with max 3 attempts + +**Context Fetching** (internal/fetcher/fetcher.go): +- HTML to plain text extraction (strips tags, extracts text nodes) +- 30-second timeout per URL +- Batch fetching fails if any URL fails + +### Important Implementation Details + +**MIME Email Parsing** (internal/imap/imap.go:287-510): +- `extractMessageContent` handles multipart and single-part messages +- Content-Transfer-Encoding support: quoted-printable +- `cleanMessageContent` has a `performHeaderStripping` flag (true for fallback, false for parsed content) +- Extensive debug logging for troubleshooting email parsing issues + +**Folder Management** (internal/imap/imap.go:512-663): +- `ensureFolder` detects Gmail vs generic IMAP servers +- Gmail: Uses "/" delimiter, skips CREATE for system folders like "[Gmail]/Drafts" +- Generic IMAP: Lists mailboxes to detect delimiter, creates folders if missing +- Has timeout protection (30s for LIST, 35s overall) +- Includes panic recovery + +**Logging Strategy**: +- All modules use `internal/logger` wrapper around logrus +- Structured logging with fields (logrus.Fields) +- Debug level logs include detailed information (e.g., email content lengths, folder paths) +- Recent commits show enhanced logging for troubleshooting email extraction + +## Development Guidelines + +### Error Handling +- Use defer/recover for panic protection in critical IMAP operations (SaveDraft, MarkAsProcessed, ensureFolder) +- Retry mechanisms in AI client (max 3 attempts) +- IMAP connection auto-reconnection on failure +- Continue processing remaining emails if one fails + +### Adding New Features +- **New AI models**: Update config.yaml model parameter (passed to OpenRouter API) +- **New context sources**: Add URLs to config.yaml context.urls array +- **New IMAP operations**: Add methods to IMAPClient struct, ensure connection handling +- **Email format changes**: Modify draft template in SaveDraft (line 242-266) + +### Testing Considerations +- Email parsing logic is complex (multipart, encoding) - test with various email formats +- Gmail behavior differs from generic IMAP servers +- Test with different folder delimiters ("/" vs ".") +- Language detection affects response language + +## Dependencies + +Core dependencies (go.mod): +- `github.com/emersion/go-imap` - IMAP client library +- `github.com/sirupsen/logrus` - Structured logging +- `golang.org/x/net/html` - HTML parsing for context extraction +- `gopkg.in/yaml.v3` - Configuration file parsing + +## Common Issues + +**IMAP Folder Issues**: +- Check logs for delimiter detection (ensureFolder) +- Gmail requires special handling for system folders +- Folder names are case-sensitive + +**Email Content Extraction**: +- Multi-part emails may require different handling +- Check Content-Transfer-Encoding header +- Recent fixes improved header stripping and content cleaning + +**AI Response Issues**: +- Verify API key is correctly loaded from environment/file +- Check model name is valid for OpenRouter +- Response must be raw HTML (system prompt enforces this) diff --git a/cmd/paraclub-ai-mailer/main.go b/cmd/paraclub-ai-mailer/main.go index 1a28588..ab9d94f 100644 --- a/cmd/paraclub-ai-mailer/main.go +++ b/cmd/paraclub-ai-mailer/main.go @@ -136,6 +136,18 @@ func processEmails(imapClient *imap.IMAPClient, fetcher *fetcher.Fetcher, aiProc } logger.WithField("responseLength", len(response)).Debug("Generated AI response") + // Mark as AI-processed immediately to prevent reprocessing if subsequent steps fail + if err := imapClient.MarkAsAIProcessed(email); err != nil { + logger.WithFields(logrus.Fields{ + "subject": email.Subject, + "error": err, + }).Warn("Failed to mark email as AI-processed, continuing with draft save") + // Continue anyway - if this fails, worst case is we might reprocess + // But we want to try to save the draft we just generated + } else { + logger.Debug("Marked email as AI-processed") + } + // Save as draft if err := imapClient.SaveDraft(email, response); err != nil { logger.WithFields(logrus.Fields{ diff --git a/internal/imap/imap.go b/internal/imap/imap.go index 91bcb22..6daf25e 100644 --- a/internal/imap/imap.go +++ b/internal/imap/imap.go @@ -18,6 +18,9 @@ import ( "github.com/sirupsen/logrus" ) +// Custom IMAP flag to mark emails as AI-processed +const AIProcessedFlag = "$AIProcessed" + type IMAPClient struct { client *client.Client config config.IMAPConfig @@ -137,9 +140,9 @@ func (ic *IMAPClient) FetchUnprocessedEmails() ([]Email, error) { return nil, nil } - // Get all messages in the inbox that haven't been seen yet + // Get all messages in the inbox that haven't been seen yet and haven't been AI-processed criteria := imap.NewSearchCriteria() - criteria.WithoutFlags = []string{"\\Seen"} + criteria.WithoutFlags = []string{"\\Seen", AIProcessedFlag} uids, err := ic.client.Search(criteria) if err != nil { @@ -738,6 +741,67 @@ func (ic *IMAPClient) MarkAsProcessed(email Email) (err error) { return nil } +// MarkAsAIProcessed marks an email with a custom flag to indicate AI has processed it +// This prevents reprocessing the same email if subsequent operations (SaveDraft, MarkAsProcessed) fail +func (ic *IMAPClient) MarkAsAIProcessed(email Email) (err error) { + defer func() { + if r := recover(); r != nil { + logger.WithFields(logrus.Fields{ + "panic": r, + "emailSubject": email.Subject, + }).Error("Panic occurred in MarkAsAIProcessed") + err = fmt.Errorf("panic in MarkAsAIProcessed: %v", r) + } + }() + + if err := ic.ensureConnection(); err != nil { + return fmt.Errorf("failed to ensure IMAP connection: %v", err) + } + + // Select source mailbox + _, err = ic.client.Select(ic.config.MailboxIn, false) + if err != nil { + return fmt.Errorf("failed to select source mailbox: %v", err) + } + + // Find the email by Message-Id + criteria := imap.NewSearchCriteria() + criteria.Header.Set("Message-Id", email.ID) + + uids, err := ic.client.Search(criteria) + if err != nil { + return fmt.Errorf("search failed: %v", err) + } + + if len(uids) == 0 { + return fmt.Errorf("email not found") + } + + // Mark with AI-processed flag + seqSet := new(imap.SeqSet) + seqSet.AddNum(uids...) + + item := imap.FormatFlagsOp(imap.AddFlags, true) + flags := []interface{}{AIProcessedFlag} + if err := ic.client.Store(seqSet, item, flags, nil); err != nil { + // If custom flags are not supported, fall back to marking as \Seen + logger.WithFields(logrus.Fields{ + "error": err, + "subject": email.Subject, + }).Warn("Failed to set custom AI-processed flag, falling back to \\Seen flag") + + flags = []interface{}{imap.SeenFlag} + if err := ic.client.Store(seqSet, item, flags, nil); err != nil { + return fmt.Errorf("failed to mark message with fallback flag: %v", err) + } + logger.WithField("subject", email.Subject).Info("Marked email as \\Seen (custom flag not supported)") + return nil + } + + logger.WithField("subject", email.Subject).Debug("Successfully marked email with AI-processed flag") + return nil +} + func (ic *IMAPClient) Close() error { if ic.client != nil { ic.client.Logout()