feat: aso add label for ai processed

CLAUDE.md

@@ -0,0 +1,165 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+ParaClub AI Mailer is an automated email response system written in Go that:
+- Fetches emails via IMAP from a configured mailbox
+- Gathers context from configured URLs (HTML content extraction)
+- Uses OpenRouter AI API to generate context-aware email replies
+- Saves AI-generated responses as email drafts
+- Processes emails sequentially with configurable polling intervals
+
+The system is designed as a long-running service with graceful shutdown handling.
+
+## Build and Run Commands
+
+```bash
+# Install dependencies
+go mod download
+
+# Build the application
+go build -o paraclub-ai-mailer ./cmd/paraclub-ai-mailer
+
+# Run with default config (config.yaml in current directory)
+./paraclub-ai-mailer
+
+# Run with custom config path
+./paraclub-ai-mailer -config /path/to/config.yaml
+
+# Run tests (if any exist)
+go test ./...
+
+# Format code
+go fmt ./...
+
+# Vet code for issues
+go vet ./...
+```
+
+## Configuration
+
+The application requires a `config.yaml` file. Use `config.yaml.example` as a template.
+
+Key configuration sections:
+- **imap**: Server details, credentials, mailbox folders
+- **ai**: OpenRouter API key, model selection, temperature, max tokens
+- **context**: List of URLs to fetch as context for AI
+- **polling**: Email check interval (e.g., "5m")
+- **logging**: Log level and file path
+
+Credentials support three formats:
+- Direct value: `password: "mypassword"`
+- Environment variable: `password: "${IMAP_PASSWORD}"`
+- File reference: `password: "file:///path/to/password.txt"`
+
+## Code Architecture
+
+### Module Structure
+
+```
+cmd/paraclub-ai-mailer/main.go    # Entry point, orchestration
+config/config.go                   # Configuration loading and validation
+internal/
+  ├── logger/logger.go             # Centralized logging wrapper (logrus)
+  ├── imap/imap.go                 # IMAP client, email operations
+  ├── fetcher/fetcher.go           # HTML content fetching and text extraction
+  └── ai/ai.go                     # OpenRouter AI integration
+```
+
+### Key Design Patterns
+
+**Email Processing Flow** (main.go:92-172):
+1. Fetch unprocessed emails (UNSEEN flag)
+2. Fetch context from all configured URLs
+3. For each email:
+   - Detect language via AI
+   - Generate AI reply with context
+   - Save as draft with original email quoted
+   - Mark as processed (move to processed_box, mark as read)
+
+**IMAP Client** (internal/imap/imap.go):
+- Connection handling with automatic reconnection (ensureConnection, reconnect)
+- Folder path normalization with delimiter detection
+- Gmail-specific logic in ensureFolder (uses "/" delimiter, handles system folders)
+- Email content extraction handles multipart messages and quoted-printable encoding
+- Draft formatting includes HTML response + quoted original message
+- Panic recovery in SaveDraft and MarkAsProcessed
+
+**AI Integration** (internal/ai/ai.go):
+- Two-stage process: language detection, then reply generation
+- Language-aware responses (system prompt includes detected language)
+- Output must be raw HTML (not markdown-wrapped)
+- Retry logic with max 3 attempts
+
+**Context Fetching** (internal/fetcher/fetcher.go):
+- HTML to plain text extraction (strips tags, extracts text nodes)
+- 30-second timeout per URL
+- Batch fetching fails if any URL fails
+
+### Important Implementation Details
+
+**MIME Email Parsing** (internal/imap/imap.go:287-510):
+- `extractMessageContent` handles multipart and single-part messages
+- Content-Transfer-Encoding support: quoted-printable
+- `cleanMessageContent` has a `performHeaderStripping` flag (true for fallback, false for parsed content)
+- Extensive debug logging for troubleshooting email parsing issues
+
+**Folder Management** (internal/imap/imap.go:512-663):
+- `ensureFolder` detects Gmail vs generic IMAP servers
+- Gmail: Uses "/" delimiter, skips CREATE for system folders like "[Gmail]/Drafts"
+- Generic IMAP: Lists mailboxes to detect delimiter, creates folders if missing
+- Has timeout protection (30s for LIST, 35s overall)
+- Includes panic recovery
+
+**Logging Strategy**:
+- All modules use `internal/logger` wrapper around logrus
+- Structured logging with fields (logrus.Fields)
+- Debug level logs include detailed information (e.g., email content lengths, folder paths)
+- Recent commits show enhanced logging for troubleshooting email extraction
+
+## Development Guidelines
+
+### Error Handling
+- Use defer/recover for panic protection in critical IMAP operations (SaveDraft, MarkAsProcessed, ensureFolder)
+- Retry mechanisms in AI client (max 3 attempts)
+- IMAP connection auto-reconnection on failure
+- Continue processing remaining emails if one fails
+
+### Adding New Features
+- **New AI models**: Update config.yaml model parameter (passed to OpenRouter API)
+- **New context sources**: Add URLs to config.yaml context.urls array
+- **New IMAP operations**: Add methods to IMAPClient struct, ensure connection handling
+- **Email format changes**: Modify draft template in SaveDraft (line 242-266)
+
+### Testing Considerations
+- Email parsing logic is complex (multipart, encoding) - test with various email formats
+- Gmail behavior differs from generic IMAP servers
+- Test with different folder delimiters ("/" vs ".")
+- Language detection affects response language
+
+## Dependencies
+
+Core dependencies (go.mod):
+- `github.com/emersion/go-imap` - IMAP client library
+- `github.com/sirupsen/logrus` - Structured logging
+- `golang.org/x/net/html` - HTML parsing for context extraction
+- `gopkg.in/yaml.v3` - Configuration file parsing
+
+## Common Issues
+
+**IMAP Folder Issues**:
+- Check logs for delimiter detection (ensureFolder)
+- Gmail requires special handling for system folders
+- Folder names are case-sensitive
+
+**Email Content Extraction**:
+- Multi-part emails may require different handling
+- Check Content-Transfer-Encoding header
+- Recent fixes improved header stripping and content cleaning
+
+**AI Response Issues**:
+- Verify API key is correctly loaded from environment/file
+- Check model name is valid for OpenRouter
+- Response must be raw HTML (system prompt enforces this)

cmd/paraclub-ai-mailer/main.go

@@ -136,6 +136,18 @@ func processEmails(imapClient *imap.IMAPClient, fetcher *fetcher.Fetcher, aiProc
 		}
 		logger.WithField("responseLength", len(response)).Debug("Generated AI response")
 
+		// Mark as AI-processed immediately to prevent reprocessing if subsequent steps fail
+		if err := imapClient.MarkAsAIProcessed(email); err != nil {
+			logger.WithFields(logrus.Fields{
+				"subject": email.Subject,
+				"error":   err,
+			}).Warn("Failed to mark email as AI-processed, continuing with draft save")
+			// Continue anyway - if this fails, worst case is we might reprocess
+			// But we want to try to save the draft we just generated
+		} else {
+			logger.Debug("Marked email as AI-processed")
+		}
+
 		// Save as draft
 		if err := imapClient.SaveDraft(email, response); err != nil {
 			logger.WithFields(logrus.Fields{

internal/imap/imap.go

@@ -18,6 +18,9 @@ import (
 	"github.com/sirupsen/logrus"
 )
 
+// Custom IMAP flag to mark emails as AI-processed
+const AIProcessedFlag = "$AIProcessed"
+
 type IMAPClient struct {
 	client *client.Client
 	config config.IMAPConfig
@@ -137,9 +140,9 @@ func (ic *IMAPClient) FetchUnprocessedEmails() ([]Email, error) {
 		return nil, nil
 	}
 
-	// Get all messages in the inbox that haven't been seen yet
+	// Get all messages in the inbox that haven't been seen yet and haven't been AI-processed
 	criteria := imap.NewSearchCriteria()
-	criteria.WithoutFlags = []string{"\\Seen"}
+	criteria.WithoutFlags = []string{"\\Seen", AIProcessedFlag}
 
 	uids, err := ic.client.Search(criteria)
 	if err != nil {
@@ -738,6 +741,67 @@ func (ic *IMAPClient) MarkAsProcessed(email Email) (err error) {
 	return nil
 }
 
+// MarkAsAIProcessed marks an email with a custom flag to indicate AI has processed it
+// This prevents reprocessing the same email if subsequent operations (SaveDraft, MarkAsProcessed) fail
+func (ic *IMAPClient) MarkAsAIProcessed(email Email) (err error) {
+	defer func() {
+		if r := recover(); r != nil {
+			logger.WithFields(logrus.Fields{
+				"panic":        r,
+				"emailSubject": email.Subject,
+			}).Error("Panic occurred in MarkAsAIProcessed")
+			err = fmt.Errorf("panic in MarkAsAIProcessed: %v", r)
+		}
+	}()
+
+	if err := ic.ensureConnection(); err != nil {
+		return fmt.Errorf("failed to ensure IMAP connection: %v", err)
+	}
+
+	// Select source mailbox
+	_, err = ic.client.Select(ic.config.MailboxIn, false)
+	if err != nil {
+		return fmt.Errorf("failed to select source mailbox: %v", err)
+	}
+
+	// Find the email by Message-Id
+	criteria := imap.NewSearchCriteria()
+	criteria.Header.Set("Message-Id", email.ID)
+
+	uids, err := ic.client.Search(criteria)
+	if err != nil {
+		return fmt.Errorf("search failed: %v", err)
+	}
+
+	if len(uids) == 0 {
+		return fmt.Errorf("email not found")
+	}
+
+	// Mark with AI-processed flag
+	seqSet := new(imap.SeqSet)
+	seqSet.AddNum(uids...)
+
+	item := imap.FormatFlagsOp(imap.AddFlags, true)
+	flags := []interface{}{AIProcessedFlag}
+	if err := ic.client.Store(seqSet, item, flags, nil); err != nil {
+		// If custom flags are not supported, fall back to marking as \Seen
+		logger.WithFields(logrus.Fields{
+			"error":   err,
+			"subject": email.Subject,
+		}).Warn("Failed to set custom AI-processed flag, falling back to \\Seen flag")
+
+		flags = []interface{}{imap.SeenFlag}
+		if err := ic.client.Store(seqSet, item, flags, nil); err != nil {
+			return fmt.Errorf("failed to mark message with fallback flag: %v", err)
+		}
+		logger.WithField("subject", email.Subject).Info("Marked email as \\Seen (custom flag not supported)")
+		return nil
+	}
+
+	logger.WithField("subject", email.Subject).Debug("Successfully marked email with AI-processed flag")
+	return nil
+}
+
 func (ic *IMAPClient) Close() error {
 	if ic.client != nil {
 		ic.client.Logout()