feat: change max token handling

.gitignore

@@ -2,7 +2,7 @@
 config.yaml
 
 # Binary
-paraclub-ai-mailer
+./paraclub-ai-mailer
 
 # Log files
 *.log

CLAUDE.md

@@ -0,0 +1,165 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+ParaClub AI Mailer is an automated email response system written in Go that:
+- Fetches emails via IMAP from a configured mailbox
+- Gathers context from configured URLs (HTML content extraction)
+- Uses OpenRouter AI API to generate context-aware email replies
+- Saves AI-generated responses as email drafts
+- Processes emails sequentially with configurable polling intervals
+
+The system is designed as a long-running service with graceful shutdown handling.
+
+## Build and Run Commands
+
+```bash
+# Install dependencies
+go mod download
+
+# Build the application
+go build -o paraclub-ai-mailer ./cmd/paraclub-ai-mailer
+
+# Run with default config (config.yaml in current directory)
+./paraclub-ai-mailer
+
+# Run with custom config path
+./paraclub-ai-mailer -config /path/to/config.yaml
+
+# Run tests (if any exist)
+go test ./...
+
+# Format code
+go fmt ./...
+
+# Vet code for issues
+go vet ./...
+```
+
+## Configuration
+
+The application requires a `config.yaml` file. Use `config.yaml.example` as a template.
+
+Key configuration sections:
+- **imap**: Server details, credentials, mailbox folders
+- **ai**: OpenRouter API key, model selection, temperature, max tokens
+- **context**: List of URLs to fetch as context for AI
+- **polling**: Email check interval (e.g., "5m")
+- **logging**: Log level and file path
+
+Credentials support three formats:
+- Direct value: `password: "mypassword"`
+- Environment variable: `password: "${IMAP_PASSWORD}"`
+- File reference: `password: "file:///path/to/password.txt"`
+
+## Code Architecture
+
+### Module Structure
+
+```
+cmd/paraclub-ai-mailer/main.go    # Entry point, orchestration
+config/config.go                   # Configuration loading and validation
+internal/
+  ├── logger/logger.go             # Centralized logging wrapper (logrus)
+  ├── imap/imap.go                 # IMAP client, email operations
+  ├── fetcher/fetcher.go           # HTML content fetching and text extraction
+  └── ai/ai.go                     # OpenRouter AI integration
+```
+
+### Key Design Patterns
+
+**Email Processing Flow** (main.go:92-172):
+1. Fetch unprocessed emails (UNSEEN flag)
+2. Fetch context from all configured URLs
+3. For each email:
+   - Detect language via AI
+   - Generate AI reply with context
+   - Save as draft with original email quoted
+   - Mark as processed (move to processed_box, mark as read)
+
+**IMAP Client** (internal/imap/imap.go):
+- Connection handling with automatic reconnection (ensureConnection, reconnect)
+- Folder path normalization with delimiter detection
+- Gmail-specific logic in ensureFolder (uses "/" delimiter, handles system folders)
+- Email content extraction handles multipart messages and quoted-printable encoding
+- Draft formatting includes HTML response + quoted original message
+- Panic recovery in SaveDraft and MarkAsProcessed
+
+**AI Integration** (internal/ai/ai.go):
+- Two-stage process: language detection, then reply generation
+- Language-aware responses (system prompt includes detected language)
+- Output must be raw HTML (not markdown-wrapped)
+- Retry logic with max 3 attempts
+
+**Context Fetching** (internal/fetcher/fetcher.go):
+- HTML to plain text extraction (strips tags, extracts text nodes)
+- 30-second timeout per URL
+- Batch fetching fails if any URL fails
+
+### Important Implementation Details
+
+**MIME Email Parsing** (internal/imap/imap.go:287-510):
+- `extractMessageContent` handles multipart and single-part messages
+- Content-Transfer-Encoding support: quoted-printable
+- `cleanMessageContent` has a `performHeaderStripping` flag (true for fallback, false for parsed content)
+- Extensive debug logging for troubleshooting email parsing issues
+
+**Folder Management** (internal/imap/imap.go:512-663):
+- `ensureFolder` detects Gmail vs generic IMAP servers
+- Gmail: Uses "/" delimiter, skips CREATE for system folders like "[Gmail]/Drafts"
+- Generic IMAP: Lists mailboxes to detect delimiter, creates folders if missing
+- Has timeout protection (30s for LIST, 35s overall)
+- Includes panic recovery
+
+**Logging Strategy**:
+- All modules use `internal/logger` wrapper around logrus
+- Structured logging with fields (logrus.Fields)
+- Debug level logs include detailed information (e.g., email content lengths, folder paths)
+- Recent commits show enhanced logging for troubleshooting email extraction
+
+## Development Guidelines
+
+### Error Handling
+- Use defer/recover for panic protection in critical IMAP operations (SaveDraft, MarkAsProcessed, ensureFolder)
+- Retry mechanisms in AI client (max 3 attempts)
+- IMAP connection auto-reconnection on failure
+- Continue processing remaining emails if one fails
+
+### Adding New Features
+- **New AI models**: Update config.yaml model parameter (passed to OpenRouter API)
+- **New context sources**: Add URLs to config.yaml context.urls array
+- **New IMAP operations**: Add methods to IMAPClient struct, ensure connection handling
+- **Email format changes**: Modify draft template in SaveDraft (line 242-266)
+
+### Testing Considerations
+- Email parsing logic is complex (multipart, encoding) - test with various email formats
+- Gmail behavior differs from generic IMAP servers
+- Test with different folder delimiters ("/" vs ".")
+- Language detection affects response language
+
+## Dependencies
+
+Core dependencies (go.mod):
+- `github.com/emersion/go-imap` - IMAP client library
+- `github.com/sirupsen/logrus` - Structured logging
+- `golang.org/x/net/html` - HTML parsing for context extraction
+- `gopkg.in/yaml.v3` - Configuration file parsing
+
+## Common Issues
+
+**IMAP Folder Issues**:
+- Check logs for delimiter detection (ensureFolder)
+- Gmail requires special handling for system folders
+- Folder names are case-sensitive
+
+**Email Content Extraction**:
+- Multi-part emails may require different handling
+- Check Content-Transfer-Encoding header
+- Recent fixes improved header stripping and content cleaning
+
+**AI Response Issues**:
+- Verify API key is correctly loaded from environment/file
+- Check model name is valid for OpenRouter
+- Response must be raw HTML (system prompt enforces this)

README.md

@@ -42,6 +42,22 @@ go build -o paraclub-ai-mailer ./cmd/paraclub-ai-mailer
 
 The application uses a YAML configuration file (`config.yaml`) with the following structure:
 
+### IMAP Settings
+- `server`: IMAP server hostname (e.g., "imap.gmail.com")
+- `port`: IMAP port (typically 993 for TLS)
+- `username`: Your email address
+- `password`: Email password (use ${IMAP_PASSWORD} to load from environment or file:///path/to/password.txt to read from file)
+- `mailbox_in`: Mailbox to check for new emails (e.g., "INBOX")
+- `draft_box`: Folder to save AI-generated drafts (e.g., "Drafts")
+- `processed_box`: Folder to move processed emails to (they will also be marked as read)
+- `use_tls`: Whether to use TLS for connection (recommended: true)
+
+### AI Settings
+- `openrouter_api_key`: Your OpenRouter API key (use ${OPENROUTER_API_KEY} to load from environment or file:///path/to/key.txt to read from file)
+- `model`: The AI model to use (e.g., "anthropic/claude-2")
+- `temperature`: Response randomness (0.0-1.0)
+- `max_tokens`: Maximum response length
+
 ```yaml
 imap:
   server: "imap.example.com"
@@ -111,7 +127,7 @@ Or specify a custom config file path:
    - Retrieves HTML content from configured URLs
    - Sends combined content to OpenRouter AI for response generation
    - Saves the AI-generated response as a draft email
-   - Marks the original email as processed
+   - Marks the original email as read and moves it to the configured processed folder
 
 ## Logging
 

cmd/paraclub-ai-mailer/main.go

@@ -0,0 +1,245 @@
+package main
+
+import (
+	"flag"
+	"os"
+	"os/signal"
+	"syscall"
+	"time"
+
+	"paraclub-ai-mailer/config"
+	"paraclub-ai-mailer/internal/ai"
+	"paraclub-ai-mailer/internal/fetcher"
+	"paraclub-ai-mailer/internal/imap"
+	"paraclub-ai-mailer/internal/logger"
+	"paraclub-ai-mailer/internal/tokens"
+
+	"github.com/sirupsen/logrus"
+)
+
+func main() {
+	configPath := flag.String("config", "config.yaml", "Path to configuration file")
+	flag.Parse()
+
+	// Load configuration
+	logger.Debug("Loading configuration from:", *configPath)
+	cfg, err := config.Load(*configPath)
+	if err != nil {
+		logger.WithError(err).Error("Failed to load configuration")
+		panic(err)
+	}
+	logger.WithFields(logrus.Fields{
+		"logLevel": cfg.Logging.Level,
+		"logFile":  cfg.Logging.FilePath,
+	}).Debug("Configuration loaded successfully")
+
+	// Initialize logger
+	if err := logger.Init(cfg.Logging.Level, cfg.Logging.FilePath); err != nil {
+		logger.WithError(err).Error("Failed to initialize logger")
+		panic(err)
+	}
+	logger.Debug("Logger initialized successfully")
+
+	// Initialize components
+	logger.WithFields(logrus.Fields{
+		"server":    cfg.IMAP.Server,
+		"port":      cfg.IMAP.Port,
+		"username":  cfg.IMAP.Username,
+		"mailboxIn": cfg.IMAP.MailboxIn,
+	}).Debug("Initializing IMAP client")
+
+	imapClient, err := imap.New(cfg.IMAP)
+	if err != nil {
+		logger.WithError(err).Error("Failed to initialize IMAP client")
+		os.Exit(1)
+	}
+	defer imapClient.Close()
+
+	fetcher := fetcher.New()
+	aiProcessor := ai.New(cfg.AI)
+
+	tokenCounter, err := tokens.New()
+	if err != nil {
+		logger.WithError(err).Error("Failed to initialize token counter")
+		os.Exit(1)
+	}
+	logger.Debug("All components initialized successfully")
+
+	// Setup signal handling for graceful shutdown
+	sigChan := make(chan os.Signal, 1)
+	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
+	done := make(chan bool, 1)
+	logger.Debug("Signal handlers configured")
+
+	// Start processing loop
+	logger.WithField("pollingInterval", cfg.Polling.Interval).Info("Starting email processing loop")
+	go func() {
+		ticker := time.NewTicker(cfg.Polling.Interval)
+		defer ticker.Stop()
+		for {
+			select {
+			case <-done:
+				logger.Debug("Received shutdown signal in processing loop")
+				return
+			case <-ticker.C:
+				logger.Debug("Processing tick started")
+				processEmails(imapClient, fetcher, aiProcessor, tokenCounter, cfg)
+				logger.Debug("Processing tick completed")
+			}
+		}
+	}()
+
+	// Wait for shutdown signal
+	sig := <-sigChan
+	logger.WithField("signal", sig.String()).Info("Received shutdown signal")
+	done <- true
+	logger.Info("Application shutdown complete")
+}
+
+func processEmails(imapClient *imap.IMAPClient, fetcher *fetcher.Fetcher, aiProcessor *ai.AI, tokenCounter *tokens.TokenCounter, cfg *config.Config) {
+	logger.Debug("Starting email processing cycle")
+
+	// Fetch unprocessed emails
+	emails, err := imapClient.FetchUnprocessedEmails()
+	if err != nil {
+		logger.WithError(err).Error("Failed to fetch emails")
+		return
+	}
+
+	logger.WithField("emailCount", len(emails)).Debug("Fetched unprocessed emails")
+
+	if len(emails) == 0 {
+		logger.Debug("No new emails to process")
+		return
+	}
+
+	// Fetch context from configured URLs
+	logger.WithField("urlCount", len(cfg.Context.URLs)).Debug("Fetching context from configured URLs")
+	contextContent, err := fetcher.FetchAllURLs(cfg.Context.URLs)
+	if err != nil {
+		logger.WithError(err).Error("Failed to fetch context content")
+		return
+	}
+	logger.WithField("contextCount", len(contextContent)).Debug("Successfully fetched context content")
+
+	// Process each email
+	var processedCount, errorCount, skippedCount int
+	for _, email := range emails {
+		logger.WithFields(logrus.Fields{
+			"subject":   email.Subject,
+			"from":      email.From,
+			"messageId": email.ID,
+		}).Info("Processing email")
+
+		// Extract clean email content (removes attachments, MIME boundaries, headers, converts HTML to text)
+		cleanEmailContent := imap.ExtractMessageContent(email.Body)
+
+		logger.WithFields(logrus.Fields{
+			"subject":    email.Subject,
+			"cleanSize":  len(cleanEmailContent),
+		}).Debug("Extracted clean email content")
+
+		// Calculate token count for validation (use English as default to avoid API call)
+		// Language detection will happen during actual GenerateReply
+		systemPrompt := aiProcessor.BuildSystemPrompt("English")
+		userPrompt := aiProcessor.BuildUserPrompt(contextContent, cleanEmailContent)
+		estimatedTokens := tokenCounter.EstimateFullRequest(systemPrompt, userPrompt)
+
+		logger.WithFields(logrus.Fields{
+			"subject":         email.Subject,
+			"estimatedTokens": estimatedTokens,
+			"maxTokens":       cfg.Processing.MaxTokens,
+		}).Debug("Calculated token estimate for email")
+
+		// Check token limit
+		if cfg.Processing.MaxTokens > 0 && estimatedTokens > cfg.Processing.MaxTokens {
+			logger.WithFields(logrus.Fields{
+				"subject":         email.Subject,
+				"from":            email.From,
+				"estimatedTokens": estimatedTokens,
+				"maxTokens":       cfg.Processing.MaxTokens,
+			}).Warn("Email exceeds token limit, marking as AI-processed but keeping in inbox")
+
+			skippedCount++
+
+			// Mark as AI-processed to prevent reprocessing, but DON'T move the email
+			if markErr := imapClient.MarkAsAIProcessed(email); markErr != nil {
+				logger.WithFields(logrus.Fields{
+					"subject": email.Subject,
+					"error":   markErr,
+				}).Error("Failed to mark oversized email as AI-processed")
+			} else {
+				logger.WithField("subject", email.Subject).Info("Marked oversized email as AI-processed (email remains in inbox)")
+			}
+			continue
+		}
+
+		// Generate AI response with clean content
+		response, err := aiProcessor.GenerateReply(cleanEmailContent, contextContent)
+		if err != nil {
+			logger.WithFields(logrus.Fields{
+				"subject": email.Subject,
+				"error":   err,
+			}).Error("Failed to generate reply")
+			errorCount++
+
+			// Mark as AI-processed even on failure to prevent infinite retry loop
+			if markErr := imapClient.MarkAsAIProcessed(email); markErr != nil {
+				logger.WithFields(logrus.Fields{
+					"subject": email.Subject,
+					"error":   markErr,
+				}).Error("Failed to mark failed email as AI-processed")
+			} else {
+				logger.WithField("subject", email.Subject).Info("Marked failed email as AI-processed to prevent reprocessing")
+			}
+			continue
+		}
+		logger.WithField("responseLength", len(response)).Debug("Generated AI response")
+
+		// Mark as AI-processed immediately to prevent reprocessing if subsequent steps fail
+		if err := imapClient.MarkAsAIProcessed(email); err != nil {
+			logger.WithFields(logrus.Fields{
+				"subject": email.Subject,
+				"error":   err,
+			}).Warn("Failed to mark email as AI-processed, continuing with draft save")
+			// Continue anyway - if this fails, worst case is we might reprocess
+			// But we want to try to save the draft we just generated
+		} else {
+			logger.Debug("Marked email as AI-processed")
+		}
+
+		// Save as draft
+		if err := imapClient.SaveDraft(email, response); err != nil {
+			logger.WithFields(logrus.Fields{
+				"subject": email.Subject,
+				"error":   err,
+			}).Error("Failed to save draft")
+			errorCount++
+			continue
+		}
+		logger.Debug("Saved response as draft")
+
+		// Mark email as processed
+		if err := imapClient.MarkAsProcessed(email); err != nil {
+			logger.WithFields(logrus.Fields{
+				"subject": email.Subject,
+				"error":   err,
+			}).Error("Failed to mark email as processed")
+			errorCount++
+			continue
+		}
+
+		processedCount++
+		logger.WithFields(logrus.Fields{
+			"subject": email.Subject,
+			"from":    email.From,
+		}).Info("Successfully processed email")
+	}
+
+	logger.WithFields(logrus.Fields{
+		"totalEmails": len(emails),
+		"processed":   processedCount,
+		"errors":      errorCount,
+		"skipped":     skippedCount,
+	}).Info("Completed email processing cycle")
+}

config.yaml.example

@@ -2,13 +2,14 @@ imap:
   server: "imap.gmail.com"  # Example for Gmail
   port: 993
   username: "your-email@gmail.com"
-  password: "${IMAP_PASSWORD}"  # Will be read from environment variable
+  password: "${IMAP_PASSWORD}"  # From environment variable, or use "file:///path/to/password.txt"
   mailbox_in: "INBOX"
   draft_box: "Drafts"
+  processed_box: "Processed"  # Folder where processed emails will be moved
   use_tls: true
 
 ai:
-  openrouter_api_key: "${OPENROUTER_API_KEY}"  # Will be read from environment variable
+  openrouter_api_key: "${OPENROUTER_API_KEY}"  # From environment variable, or use "file:///path/to/api-key.txt"
   model: "anthropic/claude-2"  # Other options: "openai/gpt-4", "google/palm-2"
   temperature: 0.7  # 0.0 to 1.0, lower for more focused responses
   max_tokens: 2000  # Adjust based on your needs and model limits
@@ -22,6 +23,13 @@ context:
 polling:
   interval: "5m"  # Examples: "30s", "1m", "1h"
 
+processing:
+  max_tokens: 12000  # Maximum total tokens for API request (system + context + email)
+                     # Recommended: 8000-12000 for most models
+                     # Set to 0 for no limit
+                     # Note: Attachments are automatically stripped before counting
+  skip_junk_emails: false  # Skip emails marked as junk/spam (not yet implemented)
+
 logging:
   level: "info"  # Options: "debug", "info", "warn", "error"
   file_path: "paraclub-ai-mailer.log"

config/config.go

@@ -11,21 +11,23 @@ import (
 )
 
 type Config struct {
-	IMAP    IMAPConfig    `yaml:"imap"`
-	AI      AIConfig      `yaml:"ai"`
-	Context ContextConfig `yaml:"context"`
-	Polling PollingConfig `yaml:"polling"`
-	Logging LoggingConfig `yaml:"logging"`
+	IMAP       IMAPConfig       `yaml:"imap"`
+	AI         AIConfig         `yaml:"ai"`
+	Context    ContextConfig    `yaml:"context"`
+	Polling    PollingConfig    `yaml:"polling"`
+	Logging    LoggingConfig    `yaml:"logging"`
+	Processing ProcessingConfig `yaml:"processing"`
 }
 
 type IMAPConfig struct {
-	Server    string `yaml:"server"`
-	Port      int    `yaml:"port"`
-	Username  string `yaml:"username"`
-	Password  string `yaml:"password"`
-	MailboxIn string `yaml:"mailbox_in"`
-	DraftBox  string `yaml:"draft_box"`
-	UseTLS    bool   `yaml:"use_tls"`
+	Server       string `yaml:"server"`
+	Port         int    `yaml:"port"`
+	Username     string `yaml:"username"`
+	Password     string `yaml:"password"`
+	MailboxIn    string `yaml:"mailbox_in"`
+	DraftBox     string `yaml:"draft_box"`
+	ProcessedBox string `yaml:"processed_box"`
+	UseTLS       bool   `yaml:"use_tls"`
 }
 
 type AIConfig struct {
@@ -48,6 +50,19 @@ type LoggingConfig struct {
 	FilePath string `yaml:"file_path"`
 }
 
+type ProcessingConfig struct {
+	MaxTokens      int  `yaml:"max_tokens"`        // Maximum total tokens for API request (0 = no limit)
+	SkipJunkEmails bool `yaml:"skip_junk_emails"`  // Skip emails marked as junk/spam
+}
+
+func readFileContent(path string) (string, error) {
+	content, err := os.ReadFile(path)
+	if err != nil {
+		return "", err
+	}
+	return strings.TrimSpace(string(content)), nil
+}
+
 func Load(path string) (*Config, error) {
 	logger.WithField("path", path).Debug("Loading configuration file")
 
@@ -64,17 +79,36 @@ func Load(path string) (*Config, error) {
 		return nil, err
 	}
 
-	// Resolve environment variables in configuration
+	// Handle IMAP password from environment or file
 	if strings.HasPrefix(config.IMAP.Password, "${") && strings.HasSuffix(config.IMAP.Password, "}") {
 		envVar := strings.TrimSuffix(strings.TrimPrefix(config.IMAP.Password, "${"), "}")
 		config.IMAP.Password = os.Getenv(envVar)
 		logger.WithField("envVar", envVar).Debug("Resolved IMAP password from environment")
+	} else if strings.HasPrefix(config.IMAP.Password, "file://") {
+		filePath := strings.TrimPrefix(config.IMAP.Password, "file://")
+		password, err := readFileContent(filePath)
+		if err != nil {
+			logger.WithError(err).Error("Failed to read IMAP password from file")
+			return nil, err
+		}
+		config.IMAP.Password = password
+		logger.WithField("path", filePath).Debug("Read IMAP password from file")
 	}
 
+	// Handle OpenRouter API key from environment or file
 	if strings.HasPrefix(config.AI.OpenRouterAPIKey, "${") && strings.HasSuffix(config.AI.OpenRouterAPIKey, "}") {
 		envVar := strings.TrimSuffix(strings.TrimPrefix(config.AI.OpenRouterAPIKey, "${"), "}")
 		config.AI.OpenRouterAPIKey = os.Getenv(envVar)
 		logger.WithField("envVar", envVar).Debug("Resolved OpenRouter API key from environment")
+	} else if strings.HasPrefix(config.AI.OpenRouterAPIKey, "file://") {
+		filePath := strings.TrimPrefix(config.AI.OpenRouterAPIKey, "file://")
+		apiKey, err := readFileContent(filePath)
+		if err != nil {
+			logger.WithError(err).Error("Failed to read OpenRouter API key from file")
+			return nil, err
+		}
+		config.AI.OpenRouterAPIKey = apiKey
+		logger.WithField("path", filePath).Debug("Read OpenRouter API key from file")
 	}
 
 	logger.WithFields(logrus.Fields{
@@ -91,6 +125,8 @@ func Load(path string) (*Config, error) {
 		"pollingInterval": config.Polling.Interval,
 		"loggingLevel":    config.Logging.Level,
 		"loggingFilePath": config.Logging.FilePath,
+		"maxTokens":       config.Processing.MaxTokens,
+		"skipJunkEmails":  config.Processing.SkipJunkEmails,
 	}).Debug("Configuration loaded successfully")
 
 	return &config, nil

go.mod

@@ -10,7 +10,10 @@ require (
 )
 
 require (
+	github.com/dlclark/regexp2 v1.10.0 // indirect
 	github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 // indirect
+	github.com/google/uuid v1.3.0 // indirect
+	github.com/pkoukk/tiktoken-go v0.1.8 // indirect
 	golang.org/x/sys v0.30.0 // indirect
 	golang.org/x/text v0.22.0 // indirect
 )

go.sum

@@ -1,6 +1,8 @@
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dlclark/regexp2 v1.10.0 h1:+/GIL799phkJqYW+3YbOd8LCcbHzT0Pbo8zl70MHsq0=
+github.com/dlclark/regexp2 v1.10.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
 github.com/emersion/go-imap v1.2.1 h1:+s9ZjMEjOB8NzZMVTM3cCenz2JrQIGGo5j1df19WjTA=
 github.com/emersion/go-imap v1.2.1/go.mod h1:Qlx1FSx2FTxjnjWpIlVNEuX+ylerZQNFE5NsmKFSejY=
 github.com/emersion/go-message v0.15.0/go.mod h1:wQUEfE+38+7EW8p8aZ96ptg6bAb1iwdgej19uXASlE4=
@@ -8,6 +10,10 @@ github.com/emersion/go-sasl v0.0.0-20200509203442-7bfe0ed36a21/go.mod h1:iL2twTe
 github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 h1:oP4q0fw+fOSWn3DfFi4EXdT+B+gTtzx8GC9xsc26Znk=
 github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6/go.mod h1:iL2twTeMvZnrg54ZoPDNfJaJaqy0xIQFuBdrLsmspwQ=
 github.com/emersion/go-textwrapper v0.0.0-20200911093747-65d896831594/go.mod h1:aqO8z8wPrjkscevZJFVE1wXJrLpC5LtJG7fqLOsPb2U=
+github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
+github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/pkoukk/tiktoken-go v0.1.8 h1:85ENo+3FpWgAACBaEUVp+lctuTcYUO7BtmfhlN/QTRo=
+github.com/pkoukk/tiktoken-go v0.1.8/go.mod h1:9NiV+i9mJKGj1rYOT+njbv+ZwA/zJxYdewGl6qVatpg=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
@@ -15,6 +21,7 @@ github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVs
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8=
 golang.org/x/net v0.35.0 h1:T5GQRQb2y08kTAByq9L4/bz8cipCdA8FbRTXewonqY8=
 golang.org/x/net v0.35.0/go.mod h1:EglIi67kWsHKlRzzVMUD93VMSWGFOMSZgxFjparz1Qk=
 golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=

internal/ai/ai.go

@@ -43,35 +43,86 @@ func New(cfg config.AIConfig) *AI {
 	}
 }
 
-func (a *AI) GenerateReply(emailContent string, contextContent map[string]string) (string, error) {
-	logger.WithFields(logrus.Fields{
-		"emailContentLength": len(emailContent),
-		"contextUrls":       len(contextContent),
-	}).Debug("Starting AI reply generation")
+// BuildSystemPrompt creates the system prompt for a given language
+// Exposed for token counting without making an API call
+func (a *AI) BuildSystemPrompt(lang string) string {
+	return fmt.Sprintf(`You are a helpful assistant who responds to emails.
+Your primary goal is to answer the user's query (found in the 'Email Body') by primarily using the information available in the 'Additional Context' and your general knowledge.
+While the 'Email Body' provides the question, your answer should be synthesized from the context and your understanding, not by directly repeating or solely relying on unverified information from the 'Email Body' itself.
 
-	// Prepare context from all URLs
+Instructions:
+- Language: Your response must be entirely in %s, regardless of the language used in the email content or context.
+- Format: CRITICAL: Your reply MUST be raw HTML. Use appropriate HTML tags for structure and styling. For example, wrap paragraphs in <p>...</p> tags and use <br> for line breaks if needed within a paragraph. Even a short sentence must be wrapped in HTML (e.g., <p>Yes.</p>).
+- Markdown: Do NOT wrap the HTML in markdown code blocks (e.g., %s).
+- Extraneous Text: Do not include a subject line. Do not include explanations, commentary, or any extra text that is not part of the direct answer.
+- Closing: Avoid generic closing statements like "If you have further questions...". Focus solely on answering the email.
+`, lang, "```html ... ```")
+}
+
+// BuildUserPrompt creates the user message with context and email content
+func (a *AI) BuildUserPrompt(contextContent map[string]string, emailContent string) string {
 	var context string
 	for url, content := range contextContent {
 		context += fmt.Sprintf("\nContext from %s:\n%s\n", url, content)
-		logger.WithFields(logrus.Fields{
-			"url":           url,
-			"contentLength": len(content),
-		}).Debug("Added context from URL")
 	}
+	return fmt.Sprintf("### Additional Context:\n%s\n\n### Email Body:\n%s", context, emailContent)
+}
 
-	// Prepare the system message and user message
-	// systemMsg := "You are a helpful assistant responding to emails. Analyze the language of the incoming email and translate the response to the same language. Format your response in HTML. Do not include any explanations or extra text - just write the email response directly in HTML format. Use appropriate HTML tags for formatting."
-  // systemMsg := "You are a helpful assistant who responds to emails. For each incoming email, first detect its language and then generate your response in the exact same language. Your reply must be written directly in HTML format using appropriate HTML tags for structure and styling. Do not include any explanations, commentary, or extra text—simply output the email response in HTML."
-  // systemMsg := "You are a helpful assistant who responds to emails. For each incoming email, first detect its language and then generate your response in the exact same language. Your reply must be written directly in HTML format using appropriate HTML tags for structure and styling. Do not include a subject line, explanations, commentary, or any extra text—simply output the email response content in HTML."
-  // systemMsg := "You are a helpful assistant who responds to emails. For every incoming email, carefully detect and confirm its language. Then, generate your email response entirely in that same language without deviation. Format your reply solely in HTML using appropriate HTML tags for structure and styling, and do not include a subject line, explanations, or any extra text. Ensure that every part of your response exactly matches the language of the incoming email."
-  systemMsg := "You are a helpful assistant who responds to emails. For every incoming email, strictly use only the email's content to detect its language and ignore any external or additional context. Then, generate your email response entirely in that same language without deviation. Format your reply solely in HTML using appropriate HTML tags for structure and styling, and do not include a subject line, explanations, commentary, or any extra text. Ensure that every part of your response exactly matches the language of the incoming email."
-	userMsg := fmt.Sprintf("Using the following context:\n%s\n\nPlease analyze the language of and generate a reply in the same language for this email:\n%s", context, emailContent)
+func (a *AI) detectLanguage(emailContent string) (string, error) {
+	logger.WithField("emailContentLength", len(emailContent)).Debug("Starting language detection")
+
+	systemMsg := "You are a language detection assistant. Analyze the provided text and respond ONLY with the language name (e.g., 'English', 'German', 'French', etc.). No other text or explanation."
+	userMsg := fmt.Sprintf("Detect the language of this text:\n\n%s", emailContent)
 
 	messages := []Message{
 		{Role: "system", Content: systemMsg},
 		{Role: "user", Content: userMsg},
 	}
 
+	response, err := a.makeAPIRequest(messages)
+	if err != nil {
+		return "", fmt.Errorf("language detection failed: %v", err)
+	}
+
+	// The response should be just the language code
+	langCode := response
+	logger.WithField("detectedLanguage", langCode).Debug("Language detection completed")
+	return langCode, nil
+}
+
+func (a *AI) GenerateReply(emailContent string, contextContent map[string]string) (string, error) {
+	logger.WithFields(logrus.Fields{
+		"emailContentLength": len(emailContent),
+		"contextUrls":        len(contextContent),
+	}).Debug("Starting AI reply generation")
+
+	// First, detect the language
+	lang, err := a.detectLanguage(emailContent)
+	if err != nil {
+		return "", err
+	}
+
+	// Build prompts using exposed methods
+	systemMsg := a.BuildSystemPrompt(lang)
+	userMsg := a.BuildUserPrompt(contextContent, emailContent)
+
+	logger.Debug("Generated system and user prompts")
+
+	messages := []Message{
+		{Role: "system", Content: systemMsg},
+		{Role: "user", Content: userMsg},
+	}
+
+	aiResponse, err := a.makeAPIRequest(messages)
+	if err != nil {
+		// Error already logged by makeAPIRequest, just propagate
+		return "", err
+	}
+	logger.WithField("rawAIResponse", aiResponse).Debug("Received raw response from AI")
+	return aiResponse, nil
+}
+
+func (a *AI) makeAPIRequest(messages []Message) (string, error) {
 	const maxRetries = 3
 	var lastErr error
 

internal/imap/imap.go

@@ -6,6 +6,7 @@ import (
 	"io"
 	"mime"
 	"mime/multipart"
+	"mime/quotedprintable"
 	"net/mail"
 	"paraclub-ai-mailer/config"
 	"paraclub-ai-mailer/internal/logger"
@@ -17,10 +18,12 @@ import (
 	"github.com/sirupsen/logrus"
 )
 
+// Custom IMAP flag to mark emails as AI-processed
+const AIProcessedFlag = "$AIProcessed"
+
 type IMAPClient struct {
-	client    *client.Client
-	config    config.IMAPConfig
-	processed map[string]bool
+	client *client.Client
+	config config.IMAPConfig
 }
 
 // MessageLiteral implements imap.Literal for draft messages
@@ -65,9 +68,8 @@ func New(cfg config.IMAPConfig) (*IMAPClient, error) {
 	}
 
 	return &IMAPClient{
-		client:    c,
-		config:    cfg,
-		processed: make(map[string]bool),
+		client: c,
+		config: cfg,
 	}, nil
 }
 
@@ -75,6 +77,7 @@ type Email struct {
 	ID      string
 	Subject string
 	From    string
+	Date    time.Time
 	Body    string
 }
 
@@ -137,8 +140,9 @@ func (ic *IMAPClient) FetchUnprocessedEmails() ([]Email, error) {
 		return nil, nil
 	}
 
+	// Get all messages in the inbox that haven't been seen yet and haven't been AI-processed
 	criteria := imap.NewSearchCriteria()
-	criteria.WithoutFlags = []string{"\\Seen", "Processed"}
+	criteria.WithoutFlags = []string{"\\Seen", AIProcessedFlag}
 
 	uids, err := ic.client.Search(criteria)
 	if err != nil {
@@ -170,11 +174,6 @@ func (ic *IMAPClient) FetchUnprocessedEmails() ([]Email, error) {
 			continue
 		}
 
-		// Skip if already processed
-		if ic.processed[msg.Envelope.MessageId] {
-			continue
-		}
-
 		// Skip if no message ID
 		if msg.Envelope.MessageId == "" {
 			continue
@@ -200,10 +199,9 @@ func (ic *IMAPClient) FetchUnprocessedEmails() ([]Email, error) {
 			ID:      msg.Envelope.MessageId,
 			Subject: msg.Envelope.Subject,
 			From:    from,
+			Date:    msg.Envelope.Date,
 			Body:    bodyBuilder.String(),
 		})
-
-		ic.processed[msg.Envelope.MessageId] = true
 	}
 
 	if err := <-done; err != nil {
@@ -213,15 +211,35 @@ func (ic *IMAPClient) FetchUnprocessedEmails() ([]Email, error) {
 	return emails, nil
 }
 
-func (ic *IMAPClient) SaveDraft(email Email, response string) error {
+func (ic *IMAPClient) SaveDraft(email Email, response string) (err error) {
+	defer func() {
+		if r := recover(); r != nil {
+			logger.WithFields(logrus.Fields{
+				"panic":        r,
+				"emailSubject": email.Subject,
+				"draftBox":     ic.config.DraftBox,
+			}).Error("Panic occurred in SaveDraft")
+			err = fmt.Errorf("panic in SaveDraft: %v", r)
+		}
+	}()
+
 	if err := ic.ensureConnection(); err != nil {
 		return fmt.Errorf("failed to ensure IMAP connection: %v", err)
 	}
 
-	_, err := ic.client.Select(ic.config.DraftBox, false)
+	// Get proper folder path and ensure it exists
+	draftFolder, err := ic.ensureFolder(ic.config.DraftBox)
 	if err != nil {
-		return fmt.Errorf("failed to select draft box: %v", err)
+		return fmt.Errorf("failed to ensure draft folder: %s, error: %v", ic.config.DraftBox, err)
 	}
+	logger.WithField("draftFolder", draftFolder).Debug("Ensured draft folder path")
+
+	logger.WithField("draftFolder", draftFolder).Debug("Attempting to select draft folder")
+	_, err = ic.client.Select(draftFolder, false)
+	if err != nil {
+		return fmt.Errorf("failed to select draft box '%s': %v", draftFolder, err)
+	}
+	logger.WithField("draftFolder", draftFolder).Debug("Successfully selected draft folder")
 
 	// Format the draft message with HTML response and original email headers + content
 	draft := fmt.Sprintf("From: %s\r\n"+
@@ -247,39 +265,46 @@ func (ic *IMAPClient) SaveDraft(email Email, response string) error {
 		response,
 		email.From,
 		email.Subject,
-		time.Now().Format("Mon, 02 Jan 2006 15:04:05 -0700"),
-		extractMessageContent(email.Body))
+		email.Date.Format("Mon, 02 Jan 2006 15:04:05 -0700"),
+		ExtractMessageContent(email.Body))
 
 	literal := &MessageLiteral{
 		content: []byte(draft),
 		pos:     0,
 	}
 
-	// Save the draft
+	// Save the draft to the proper folder path
 	flags := []string{"\\Draft"}
-	if err := ic.client.Append(ic.config.DraftBox, flags, time.Now(), literal); err != nil {
-		return fmt.Errorf("failed to append draft: %v", err)
+	logger.WithFields(logrus.Fields{
+		"draftFolder": draftFolder,
+		"flags":       flags,
+	}).Debug("Attempting to append draft")
+	if err := ic.client.Append(draftFolder, flags, time.Now(), literal); err != nil {
+		return fmt.Errorf("failed to append draft to '%s': %v", draftFolder, err)
 	}
+	logger.WithField("draftFolder", draftFolder).Debug("Successfully appended draft")
 
 	return nil
 }
 
-// extractMessageContent attempts to extract just the message content
+// ExtractMessageContent attempts to extract just the message content
 // by removing email headers and MIME boundaries
-func extractMessageContent(body string) string {
+func ExtractMessageContent(body string) string {
 	logger.WithField("bodyLength", len(body)).Debug("Starting message content extraction")
-
+	logger.WithField("rawInputBody", body).Debug("extractMessageContent: Raw input body")
 	msg, err := mail.ReadMessage(strings.NewReader(body))
 	if err != nil {
 		logger.WithFields(logrus.Fields{
 			"error":      err,
 			"bodyLength": len(body),
 		}).Debug("Failed to parse email message, falling back to simple extraction")
-		return fallbackExtractContent(body)
+		// When ReadMessage fails, the body is raw, so header stripping is needed.
+		return cleanMessageContent(fallbackExtractContent(body), true)
 	}
 
 	contentTypeHeader := msg.Header.Get("Content-Type")
 	logger.WithField("contentTypeHeader", contentTypeHeader).Debug("Got Content-Type header")
+	logger.WithField("parsedContentTypeHeader", contentTypeHeader).Debug("extractMessageContent: Parsed Content-Type header")
 
 	mediaType, params, err := mime.ParseMediaType(contentTypeHeader)
 	if err != nil {
@@ -287,149 +312,573 @@ func extractMessageContent(body string) string {
 			"error":             err,
 			"contentTypeHeader": contentTypeHeader,
 		}).Debug("Failed to parse Content-Type header, falling back to simple extraction")
-		return fallbackExtractContent(body)
+		// When ParseMediaType fails, the body is raw, so header stripping is needed.
+		return cleanMessageContent(fallbackExtractContent(body), true)
 	}
 
 	logger.WithFields(logrus.Fields{
 		"mediaType": mediaType,
 		"params":    params,
 	}).Debug("Parsed message Content-Type")
+	logger.WithFields(logrus.Fields{
+		"mediaType": mediaType,
+		"params":    params,
+	}).Debug("extractMessageContent: Parsed mediaType and params")
 
+	var content string
 	if strings.HasPrefix(mediaType, "multipart/") {
-		boundary := params["boundary"]
-		if boundary == "" {
-			logger.WithField("mediaType", mediaType).Debug("No boundary found in multipart message, falling back to simple extraction")
-			return fallbackExtractContent(body)
+		// For multipart, the handling of Content-Transfer-Encoding will be done within handleMultipartMessage for each part
+		logger.Debug("extractMessageContent: Handling as multipart message")
+		content = handleMultipartMessage(msg.Body, params["boundary"])
+		logger.WithField("contentFromMultipart", content).Debug("extractMessageContent: Content after handleMultipartMessage")
+	} else {
+		// For single part, handle Content-Transfer-Encoding here
+		var partReader io.Reader = msg.Body
+		transferEncoding := strings.ToLower(msg.Header.Get("Content-Transfer-Encoding"))
+		if transferEncoding == "quoted-printable" {
+			partReader = quotedprintable.NewReader(msg.Body)
+		}
+		// Add handling for "base64" if needed in the future
+		// else if transferEncoding == "base64" {
+		// 	partReader = base64.NewDecoder(base64.StdEncoding, msg.Body)
+		// }
+		logger.Debug("extractMessageContent: Handling as single part message")
+		content = handleSinglePartMessage(partReader)
+		logger.WithField("contentFromSinglePart", content).Debug("extractMessageContent: Content after handleSinglePartMessage")
+	}
+
+	if content == "" {
+		logger.Debug("No content extracted, falling back to simple extraction")
+		logger.Debug("extractMessageContent: No content from primary extraction, falling back.")
+		// When primary extraction yields no content, the body is raw, so header stripping is needed.
+		return cleanMessageContent(fallbackExtractContent(body), true)
+	}
+
+	// Clean up the content
+	logger.WithField("contentBeforeClean", content).Debug("extractMessageContent: Content before cleanMessageContent")
+	// When ReadMessage succeeds, 'content' is already the message body (or part body),
+	// so no further header stripping should be done.
+	content = cleanMessageContent(content, false)
+	logger.WithField("contentAfterClean", content).Debug("extractMessageContent: Content after cleanMessageContent")
+
+	logger.WithField("contentLength", len(content)).Debug("Successfully extracted and cleaned message content")
+	return content
+}
+
+func handleMultipartMessage(reader io.Reader, boundary string) string {
+	logger.WithField("boundary", boundary).Debug("handleMultipartMessage: Starting with boundary")
+	if boundary == "" {
+		logger.Debug("No boundary found in multipart message")
+		return ""
+	}
+
+	mReader := multipart.NewReader(reader, boundary)
+	var textPlainContent string
+	var textHTMLContent string
+	partIndex := 0
+
+	for {
+		part, err := mReader.NextPart()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			logger.WithError(err).Debug("Error reading multipart part")
+			return ""
 		}
 
+		contentType := part.Header.Get("Content-Type")
+		contentDisposition := part.Header.Get("Content-Disposition")
+		contentTransferEncoding := strings.ToLower(part.Header.Get("Content-Transfer-Encoding"))
+
 		logger.WithFields(logrus.Fields{
-			"mediaType": mediaType,
-			"boundary":  boundary,
-		}).Debug("Processing multipart message")
+			"partIndex":            partIndex,
+			"partContentType":      contentType,
+			"partDisposition":      contentDisposition,
+			"partTransferEncoding": contentTransferEncoding,
+		}).Debug("handleMultipartMessage: Processing part")
 
-		reader := multipart.NewReader(msg.Body, boundary)
-		var textContent string
-		partIndex := 0
+		// Skip attachments
+		if strings.HasPrefix(contentDisposition, "attachment") {
+			logger.WithFields(logrus.Fields{
+				"partIndex": partIndex,
+				"filename":  part.FileName(),
+			}).Debug("Skipping attachment part")
+			partIndex++
+			continue
+		}
 
-		for {
-			part, err := reader.NextPart()
-			if err == io.EOF {
-				logger.Debug("Finished processing all multipart parts")
-				break
-			}
-			if err != nil {
-				logger.WithFields(logrus.Fields{
-					"error":     err,
-					"partIndex": partIndex,
-				}).Debug("Error reading multipart part, falling back to simple extraction")
-				return fallbackExtractContent(body)
-			}
-
-			contentType := part.Header.Get("Content-Type")
+		// Skip non-text content types (images, videos, applications, etc.)
+		if !strings.HasPrefix(contentType, "text/plain") &&
+			!strings.HasPrefix(contentType, "text/html") {
 			logger.WithFields(logrus.Fields{
 				"partIndex":   partIndex,
 				"contentType": contentType,
-			}).Debug("Processing message part")
-
-			if strings.HasPrefix(contentType, "text/plain") {
-				buf := new(bytes.Buffer)
-				_, err := buf.ReadFrom(part)
-				if err != nil {
-					logger.WithFields(logrus.Fields{
-						"error":     err,
-						"partIndex": partIndex,
-					}).Debug("Failed to read text/plain part")
-					continue
-				}
-				textContent = buf.String()
-				logger.WithFields(logrus.Fields{
-					"partIndex":     partIndex,
-					"contentLength": len(textContent),
-				}).Debug("Successfully extracted text/plain content")
-				textContent = strings.ReplaceAll(textContent, "\n", "<br>\n")
-				textContent = strings.ReplaceAll(textContent, "\r\n", "<br>\n")
-				break
-			}
+			}).Debug("Skipping non-text content type")
 			partIndex++
+			continue
 		}
 
-		if textContent != "" {
-			logger.WithField("contentLength", len(textContent)).Debug("Successfully extracted content from multipart message")
-			return textContent
+		// Handle quoted-printable encoding
+		var partReader io.Reader = part
+		if contentTransferEncoding == "quoted-printable" {
+			partReader = quotedprintable.NewReader(part)
 		}
-		logger.Debug("No text/plain content found in multipart message, trying to read body directly")
+
+		buf := new(bytes.Buffer)
+		if _, err := buf.ReadFrom(partReader); err != nil {
+			logger.WithError(err).WithField("partIndex", partIndex).Debug("Failed to read from partReader in multipart")
+			partIndex++
+			continue
+		}
+
+		// Store text/plain and text/html separately
+		if strings.HasPrefix(contentType, "text/plain") {
+			textPlainContent = buf.String()
+			logger.WithField("textPlainLength", len(textPlainContent)).Debug("Found text/plain part")
+		} else if strings.HasPrefix(contentType, "text/html") {
+			textHTMLContent = buf.String()
+			logger.WithField("textHTMLLength", len(textHTMLContent)).Debug("Found text/html part")
+		}
+
+		partIndex++
 	}
 
+	// Prefer text/plain over text/html
+	if textPlainContent != "" {
+		logger.Debug("handleMultipartMessage: Returning text/plain content")
+		return textPlainContent
+	}
+
+	if textHTMLContent != "" {
+		logger.Debug("handleMultipartMessage: Converting text/html to plain text")
+		return htmlToPlainText(textHTMLContent)
+	}
+
+	logger.Debug("handleMultipartMessage: No text content found")
+	return ""
+}
+
+func handleSinglePartMessage(reader io.Reader) string {
+	logger.Debug("handleSinglePartMessage: Starting")
 	buf := new(bytes.Buffer)
-	_, err = buf.ReadFrom(msg.Body)
-	if err != nil {
-		logger.WithFields(logrus.Fields{
-			"error":     err,
-			"mediaType": mediaType,
-		}).Debug("Failed to read message body, falling back to simple extraction")
-		return fallbackExtractContent(body)
+	if _, err := buf.ReadFrom(reader); err != nil {
+		logger.WithError(err).Debug("Failed to read message body")
+		return ""
+	}
+	content := buf.String()
+	logger.WithField("readContent", content).Debug("handleSinglePartMessage: Content read from reader")
+	return content
+}
+
+// htmlToPlainText converts HTML content to plain text by extracting text nodes
+func htmlToPlainText(htmlContent string) string {
+	logger.WithField("htmlLength", len(htmlContent)).Debug("Converting HTML to plain text")
+
+	// Simple HTML tag stripping - removes all HTML tags and extracts text
+	var result strings.Builder
+	inTag := false
+
+	for _, char := range htmlContent {
+		switch char {
+		case '<':
+			inTag = true
+		case '>':
+			inTag = false
+		default:
+			if !inTag {
+				result.WriteRune(char)
+			}
+		}
 	}
 
-	content := buf.String()
-	content = strings.ReplaceAll(content, "\r\n", "<br>\n")
+	plainText := result.String()
+
+	// Clean up excessive whitespace
+	plainText = strings.ReplaceAll(plainText, "\r\n", "\n")
+	plainText = strings.ReplaceAll(plainText, "\r", "\n")
+
+	// Replace multiple spaces with single space
+	lines := strings.Split(plainText, "\n")
+	var cleanLines []string
+	for _, line := range lines {
+		trimmed := strings.TrimSpace(line)
+		if trimmed != "" {
+			cleanLines = append(cleanLines, trimmed)
+		}
+	}
+
+	result2 := strings.Join(cleanLines, "\n")
+	logger.WithField("plainTextLength", len(result2)).Debug("HTML converted to plain text")
+	return result2
+}
+
+func cleanMessageContent(content string, performHeaderStripping bool) string {
+	logger.WithField("inputContentLength", len(content)).Debug("cleanMessageContent: Starting")
+	logger.WithField("performHeaderStripping", performHeaderStripping).Debug("cleanMessageContent: performHeaderStripping flag")
+
+	if performHeaderStripping {
+		// Remove any remaining email headers that might be in the body
+		lines := strings.Split(content, "\n")
+		var cleanLines []string
+		headerSection := true
+
+		for _, line := range lines {
+			trimmed := strings.TrimSpace(line)
+
+			// Empty line marks the end of headers
+			if headerSection && trimmed == "" {
+				headerSection = false
+				continue
+			}
+
+			// Skip header lines
+			if headerSection && (strings.Contains(trimmed, ":") || trimmed == "") {
+				continue
+			}
+
+			// Add non-header lines
+			// This condition was originally !headerSection, but if we are past the headers,
+			// we should always add the line. If headerSection is still true here, it means
+			// it's the first line of content after potential headers were skipped.
+			cleanLines = append(cleanLines, line)
+		}
+		content = strings.Join(cleanLines, "\n")
+		logger.WithField("contentAfterHeaderStripLength", len(content)).Debug("cleanMessageContent: Content after header stripping")
+	} else {
+		logger.Debug("cleanMessageContent: Skipping header stripping")
+	}
+
+	// Convert newlines to HTML breaks for display
+	// First, normalize all newlines (\r\n or \n) to just \n
+	content = strings.ReplaceAll(content, "\r\n", "\n")
+	// Then, replace each \n with <br>\n
 	content = strings.ReplaceAll(content, "\n", "<br>\n")
-	logger.WithFields(logrus.Fields{
-		"contentLength": len(content),
-		"mediaType":     mediaType,
-	}).Debug("Successfully extracted content from message body")
-	return content
+	logger.WithField("contentAfterNewlineConversionLength", len(content)).Debug("cleanMessageContent: Content after newline conversion")
+
+	// Remove any remaining email signature markers
+	content = strings.Split(content, "\n-- ")[0]
+	content = strings.Split(content, "<br>-- ")[0]
+
+	finalContent := strings.TrimSpace(content)
+	logger.WithField("finalCleanedContentLength", len(finalContent)).Debug("cleanMessageContent: Returning final cleaned content")
+	return finalContent
 }
 
 // fallbackExtractContent is the previous implementation used as fallback
 func fallbackExtractContent(body string) string {
 	logger.WithField("bodyLength", len(body)).Debug("Using fallback content extraction method")
+	logger.WithField("rawInputBodyFallbackLength", len(body)).Debug("fallbackExtractContent: Raw input body")
+	var content string
 	parts := strings.Split(body, "\r\n\r\n")
 	if len(parts) > 1 {
-		content := strings.Join(parts[1:], "\r\n\r\n")
-		content = strings.ReplaceAll(content, "\r\n", "<br>\n")
-		content = strings.ReplaceAll(content, "\n", "<br>\n")
+		content = strings.Join(parts[1:], "\r\n\r\n")
 		logger.WithFields(logrus.Fields{
 			"contentLength": len(content),
 			"partsCount":    len(parts),
-		}).Debug("Successfully extracted content using fallback method")
-		return content
+		}).Debug("Successfully extracted content using fallback method (from parts)")
+		logger.WithField("extractedContentFallbackLength", len(content)).Debug("fallbackExtractContent: Content from splitting parts")
+	} else {
+		content = body
+		logger.WithFields(logrus.Fields{
+			"contentLength": len(content),
+			"fullBody":      true,
+		}).Debug("Using full body as content in fallback method")
+		logger.WithField("finalContentFallbackLength", len(content)).Debug("fallbackExtractContent: Final content from full body (no parts split)")
 	}
-	content := body
-	content = strings.ReplaceAll(content, "\r\n", "<br>\n")
-	content = strings.ReplaceAll(content, "\n", "<br>\n")
-	logger.WithFields(logrus.Fields{
-		"contentLength": len(content),
-		"fullBody":      true,
-	}).Debug("Using full body as content in fallback method")
+	// Newline conversion and signature stripping will be handled by cleanMessageContent,
+	// which is now called by the callers of fallbackExtractContent.
 	return content
 }
 
-func (ic *IMAPClient) MarkAsProcessed(email Email) error {
+// ensureFolder makes sure a folder exists and returns its full path using proper delimiters
+func (ic *IMAPClient) ensureFolder(folderName string) (path string, err error) {
+	defer func() {
+		if r := recover(); r != nil {
+			logger.WithFields(logrus.Fields{
+				"panic":      r,
+				"folderName": folderName,
+			}).Error("Panic occurred in ensureFolder")
+			err = fmt.Errorf("panic in ensureFolder: %v", r)
+		}
+	}()
+	logger.WithField("folderNameInput", folderName).Debug("Ensuring folder exists (ensureFolder start)")
+
+	var folderPath string
+	isGmail := strings.ToLower(ic.config.Server) == "imap.gmail.com"
+
+	if isGmail {
+		logger.Debug("ensureFolder: Detected Gmail server. Using Gmail-specific logic.")
+		// Gmail always uses '/' as a delimiter and folder names are typically as provided.
+		delimiter := "/"
+		folderPath = folderName // Gmail folder names in config should already be correct.
+		logger.WithFields(logrus.Fields{
+			"folderName":        folderName,
+			"assumedDelimiter":  delimiter,
+			"derivedFolderPath": folderPath,
+		}).Debug("ensureFolder: Gmail - folder path set")
+
+		// For Gmail, don't try to CREATE system folders like "[Gmail]/..." or "INBOX"
+		// For "INBOX/Subfolder", CREATE is fine as it creates a label.
+		if strings.HasPrefix(folderPath, "[Gmail]/") || folderPath == "INBOX" {
+			logger.WithField("folderPath", folderPath).Debug("ensureFolder: Gmail - Skipping CREATE for system folder.")
+			return folderPath, nil
+		}
+		// For other paths like "INBOX/Done" or "MyCustomLabel", attempt CREATE.
+		logger.WithField("folderPath", folderPath).Debug("ensureFolder: Gmail - Attempting to create folder (label).")
+		if createErr := ic.client.Create(folderPath); createErr != nil {
+			if !strings.Contains(strings.ToLower(createErr.Error()), "already exists") &&
+				!strings.Contains(strings.ToLower(createErr.Error()), "mailbox exists") {
+				logger.WithFields(logrus.Fields{
+					"folder": folderPath,
+					"error":  createErr,
+				}).Warn("ensureFolder: Gmail - Folder creation failed (and not 'already exists')")
+				// Unlike non-Gmail, if CREATE fails for a non-system folder, it's more likely a real issue.
+				// However, subsequent SELECT/APPEND will ultimately determine usability.
+				// For now, we'll proceed and let later operations fail if it's critical.
+			} else {
+				logger.WithFields(logrus.Fields{
+					"folder": folderPath,
+					"error":  createErr,
+				}).Debug("ensureFolder: Gmail - Folder creation failed (likely already exists).")
+			}
+		} else {
+			logger.WithField("folderPath", folderPath).Info("ensureFolder: Gmail - Successfully created folder/label or it already existed.")
+		}
+		return folderPath, nil
+
+	} else {
+		logger.Debug("ensureFolder: Non-Gmail server. Using generic logic.")
+		// Generic logic for non-Gmail servers (existing logic with timeouts)
+		logger.Debug("ensureFolder: Creating channels mailboxes and listDone")
+		mailboxes := make(chan *imap.MailboxInfo, 10)
+		listDone := make(chan error, 1)
+		logger.Debug("ensureFolder: Channels created. Attempting to start List goroutine")
+
+		go func() {
+			logger.Debug("ensureFolder: List goroutine started")
+			defer func() {
+				if r := recover(); r != nil {
+					logger.WithFields(logrus.Fields{
+						"panic":      r,
+						"folderName": folderName,
+					}).Error("Panic occurred in ensureFolder's List goroutine")
+					listDone <- fmt.Errorf("panic in List goroutine: %v", r)
+				}
+			}()
+			logger.Debug("ensureFolder: List goroutine: Entering select for client.List or timeout")
+			select {
+			case listDone <- ic.client.List("", "*", mailboxes):
+				logger.Debug("ensureFolder: List goroutine: client.List call completed and sent to listDone")
+			case <-time.After(30 * time.Second):
+				logger.Error("Timeout occurred during client.List operation in goroutine")
+				listDone <- fmt.Errorf("client.List timeout after 30 seconds in goroutine")
+			}
+			logger.Debug("ensureFolder: List goroutine finished")
+		}()
+		logger.Debug("ensureFolder: List goroutine launched.")
+
+		var delimiter string
+		var receivedMailbox bool
+		logger.Debug("ensureFolder: Attempting to range over mailboxes channel")
+		for m := range mailboxes {
+			receivedMailbox = true
+			logger.WithField("mailboxName", m.Name).Debug("ensureFolder: Received mailbox from channel")
+			delimiter = m.Delimiter
+			logger.WithField("delimiter", delimiter).Debug("ensureFolder: Got delimiter from mailbox info")
+			break
+		}
+		if !receivedMailbox {
+			logger.Debug("ensureFolder: Did not receive any mailboxes from channel (it might have been empty or closed early).")
+		}
+		logger.Debug("ensureFolder: Finished ranging over mailboxes. Attempting to read from listDone channel with timeout")
+
+		select {
+		case listErr := <-listDone:
+			if listErr != nil {
+				logger.WithError(listErr).Error("ensureFolder: Error received from listDone channel")
+				return "", fmt.Errorf("failed to list mailboxes to determine delimiter: %v", listErr)
+			}
+			logger.Debug("ensureFolder: Successfully read from listDone channel.")
+		case <-time.After(35 * time.Second):
+			logger.Error("ensureFolder: Timeout waiting for listDone channel")
+			return "", fmt.Errorf("timeout waiting for LIST operation to complete")
+		}
+
+		if delimiter == "" {
+			logger.Debug("ensureFolder: Delimiter is still empty after processing listDone and mailboxes.")
+			delimiter = "/"
+			logger.Debug("No delimiter returned by server, using fallback '/'")
+		}
+		logger.WithField("delimiter", delimiter).Debug("Determined mailbox delimiter")
+
+		if delimiter != "/" {
+			folderPath = strings.ReplaceAll(folderName, "/", delimiter)
+		} else {
+			folderPath = folderName
+		}
+		logger.WithFields(logrus.Fields{
+			"originalFolderName": folderName,
+			"serverDelimiter":    delimiter,
+			"derivedFolderPath":  folderPath,
+		}).Debug("Derived folder path using server delimiter")
+
+		logger.WithField("folderPath", folderPath).Debug("Attempting to create folder if it doesn't exist")
+		if createErr := ic.client.Create(folderPath); createErr != nil {
+			if !strings.Contains(strings.ToLower(createErr.Error()), "already exists") &&
+				!strings.Contains(strings.ToLower(createErr.Error()), "mailbox exists") {
+				logger.WithFields(logrus.Fields{
+					"folder": folderPath,
+					"error":  createErr,
+				}).Warn("Folder creation failed (and not 'already exists')")
+			} else {
+				logger.WithFields(logrus.Fields{
+					"folder": folderPath,
+					"error":  createErr,
+				}).Debug("Folder creation failed (likely already exists)")
+			}
+		} else {
+			logger.WithField("folderPath", folderPath).Info("Successfully created folder or it already existed")
+		}
+		return folderPath, nil
+	}
+}
+
+func (ic *IMAPClient) MarkAsProcessed(email Email) (err error) {
+	defer func() {
+		if r := recover(); r != nil {
+			logger.WithFields(logrus.Fields{
+				"panic":        r,
+				"emailSubject": email.Subject,
+				"processedBox": ic.config.ProcessedBox,
+			}).Error("Panic occurred in MarkAsProcessed")
+			err = fmt.Errorf("panic in MarkAsProcessed: %v", r)
+		}
+	}()
+
 	if err := ic.ensureConnection(); err != nil {
 		return fmt.Errorf("failed to ensure IMAP connection: %v", err)
 	}
 
-	_, err := ic.client.Select(ic.config.MailboxIn, false)
-	if err != nil {
-		return err
+	if ic.config.ProcessedBox == "" {
+		return fmt.Errorf("processed_box not configured")
 	}
 
+	// Get proper folder path and ensure it exists
+	processedFolder, err := ic.ensureFolder(ic.config.ProcessedBox)
+	if err != nil {
+		return fmt.Errorf("failed to ensure processed folder: %s, error: %v", ic.config.ProcessedBox, err)
+	}
+	logger.WithField("processedFolder", processedFolder).Debug("Ensured processed folder path")
+
+	// Select source mailbox
+	_, err = ic.client.Select(ic.config.MailboxIn, false)
+	if err != nil {
+		return fmt.Errorf("failed to select source mailbox: %v", err)
+	}
+
+	// Find the email by Message-Id
 	criteria := imap.NewSearchCriteria()
 	criteria.Header.Set("Message-Id", email.ID)
 
 	uids, err := ic.client.Search(criteria)
 	if err != nil {
-		return err
+		return fmt.Errorf("search failed: %v", err)
 	}
 
 	if len(uids) == 0 {
 		return fmt.Errorf("email not found")
 	}
 
+	// Move the message to processed folder
 	seqSet := new(imap.SeqSet)
 	seqSet.AddNum(uids...)
 
-	return ic.client.Store(seqSet, imap.FormatFlagsOp(imap.AddFlags, true), []interface{}{"Processed"}, nil)
+	// Mark as read before moving
+	item := imap.FormatFlagsOp(imap.AddFlags, true)
+	flags := []interface{}{imap.SeenFlag}
+	if err := ic.client.Store(seqSet, item, flags, nil); err != nil {
+		return fmt.Errorf("failed to mark message as read: %v", err)
+	}
+
+	// Copy to processed folder using the proper path
+	if err := ic.client.Copy(seqSet, processedFolder); err != nil {
+		return fmt.Errorf("failed to copy to processed folder: %v", err)
+	}
+
+	// Delete from source folder
+	item = imap.FormatFlagsOp(imap.AddFlags, true)
+	flags = []interface{}{imap.DeletedFlag}
+	if err := ic.client.Store(seqSet, item, flags, nil); err != nil {
+		return fmt.Errorf("failed to mark message as deleted: %v", err)
+	}
+
+	if err := ic.client.Expunge(nil); err != nil {
+		return fmt.Errorf("failed to expunge message: %v", err)
+	}
+
+	return nil
+}
+
+// MarkAsAIProcessed marks an email with a custom flag to indicate AI has processed it
+// This prevents reprocessing the same email if subsequent operations (SaveDraft, MarkAsProcessed) fail
+func (ic *IMAPClient) MarkAsAIProcessed(email Email) (err error) {
+	defer func() {
+		if r := recover(); r != nil {
+			logger.WithFields(logrus.Fields{
+				"panic":        r,
+				"emailSubject": email.Subject,
+			}).Error("Panic occurred in MarkAsAIProcessed")
+			err = fmt.Errorf("panic in MarkAsAIProcessed: %v", r)
+		}
+	}()
+
+	if err := ic.ensureConnection(); err != nil {
+		return fmt.Errorf("failed to ensure IMAP connection: %v", err)
+	}
+
+	// Select source mailbox
+	_, err = ic.client.Select(ic.config.MailboxIn, false)
+	if err != nil {
+		return fmt.Errorf("failed to select source mailbox: %v", err)
+	}
+
+	// Find the email by Message-Id
+	criteria := imap.NewSearchCriteria()
+	criteria.Header.Set("Message-Id", email.ID)
+
+	uids, err := ic.client.Search(criteria)
+	if err != nil {
+		return fmt.Errorf("search failed: %v", err)
+	}
+
+	if len(uids) == 0 {
+		return fmt.Errorf("email not found")
+	}
+
+	// Mark with AI-processed flag
+	seqSet := new(imap.SeqSet)
+	seqSet.AddNum(uids...)
+
+	item := imap.FormatFlagsOp(imap.AddFlags, true)
+	flags := []interface{}{AIProcessedFlag}
+	if err := ic.client.Store(seqSet, item, flags, nil); err != nil {
+		// If custom flags are not supported, fall back to marking as \Seen
+		logger.WithFields(logrus.Fields{
+			"error":   err,
+			"subject": email.Subject,
+		}).Warn("Failed to set custom AI-processed flag, falling back to \\Seen flag")
+
+		flags = []interface{}{imap.SeenFlag}
+		if err := ic.client.Store(seqSet, item, flags, nil); err != nil {
+			return fmt.Errorf("failed to mark message with fallback flag: %v", err)
+		}
+		logger.WithField("subject", email.Subject).Info("Marked email as \\Seen (custom flag not supported)")
+		return nil
+	}
+
+	logger.WithField("subject", email.Subject).Debug("Successfully marked email with AI-processed flag")
+	return nil
 }
 
 func (ic *IMAPClient) Close() error {

internal/tokens/tokens.go

@@ -0,0 +1,50 @@
+package tokens
+
+import (
+	"fmt"
+
+	"paraclub-ai-mailer/internal/logger"
+
+	"github.com/pkoukk/tiktoken-go"
+	"github.com/sirupsen/logrus"
+)
+
+type TokenCounter struct {
+	encoding *tiktoken.Tiktoken
+}
+
+// New creates a token counter with cl100k_base encoding (GPT-4/Claude compatible)
+func New() (*TokenCounter, error) {
+	enc, err := tiktoken.GetEncoding("cl100k_base")
+	if err != nil {
+		return nil, fmt.Errorf("failed to get tiktoken encoding: %w", err)
+	}
+	return &TokenCounter{encoding: enc}, nil
+}
+
+// CountString counts tokens in a single string
+func (tc *TokenCounter) CountString(text string) int {
+	tokens := tc.encoding.Encode(text, nil, nil)
+	return len(tokens)
+}
+
+// EstimateFullRequest estimates total tokens for the complete API request
+// Includes: system prompt + user prompt + message overhead
+func (tc *TokenCounter) EstimateFullRequest(systemPrompt, userPrompt string) int {
+	systemTokens := tc.CountString(systemPrompt)
+	userTokens := tc.CountString(userPrompt)
+
+	// Add overhead for message structure (~100 tokens for JSON formatting, role labels, etc.)
+	overhead := 100
+
+	total := systemTokens + userTokens + overhead
+
+	logger.WithFields(logrus.Fields{
+		"systemTokens": systemTokens,
+		"userTokens":   userTokens,
+		"overhead":     overhead,
+		"total":        total,
+	}).Debug("Token estimation breakdown")
+
+	return total
+}