Paraclub/ai-mailer
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: Paraclub:master
Branches Tags
Paraclub:master
..
compare: Paraclub:b33a0d82299fd796a7c491bf162bd400e5ee92d7
Branches Tags
Paraclub:master

1 Commits
master ... b33a0d8229

Author SHA1 Message Date
Dominik Polakovics Polakovics
b33a0d8229 Delete main 2025-05-29 16:47:24 +02:00
9 changed files with 87 additions and 626 deletions
Expand all files Collapse all files

165
CLAUDE.md
View File

@@ -1,165 +0,0 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
ParaClub AI Mailer is an automated email response system written in Go that:
- Fetches emails via IMAP from a configured mailbox
- Gathers context from configured URLs (HTML content extraction)
- Uses OpenRouter AI API to generate context-aware email replies
- Saves AI-generated responses as email drafts
- Processes emails sequentially with configurable polling intervals
The system is designed as a long-running service with graceful shutdown handling.
## Build and Run Commands
```bash
# Install dependencies
go mod download
# Build the application
go build -o paraclub-ai-mailer ./cmd/paraclub-ai-mailer
# Run with default config (config.yaml in current directory)
./paraclub-ai-mailer
# Run with custom config path
./paraclub-ai-mailer -config /path/to/config.yaml
# Run tests (if any exist)
go test ./...
# Format code
go fmt ./...
# Vet code for issues
go vet ./...
```
## Configuration
The application requires a `config.yaml` file. Use `config.yaml.example` as a template.
Key configuration sections:
- **imap**: Server details, credentials, mailbox folders
- **ai**: OpenRouter API key, model selection, temperature, max tokens
- **context**: List of URLs to fetch as context for AI
- **polling**: Email check interval (e.g., "5m")
- **logging**: Log level and file path
Credentials support three formats:
- Direct value: `password: "mypassword"`
- Environment variable: `password: "${IMAP_PASSWORD}"`
- File reference: `password: "file:///path/to/password.txt"`
## Code Architecture
### Module Structure
```
cmd/paraclub-ai-mailer/main.go # Entry point, orchestration
config/config.go # Configuration loading and validation
internal/
├── logger/logger.go # Centralized logging wrapper (logrus)
├── imap/imap.go # IMAP client, email operations
├── fetcher/fetcher.go # HTML content fetching and text extraction
└── ai/ai.go # OpenRouter AI integration
```
### Key Design Patterns
**Email Processing Flow** (main.go:92-172):
1. Fetch unprocessed emails (UNSEEN flag)
2. Fetch context from all configured URLs
3. For each email:
- Detect language via AI
- Generate AI reply with context
- Save as draft with original email quoted
- Mark as processed (move to processed_box, mark as read)
**IMAP Client** (internal/imap/imap.go):
- Connection handling with automatic reconnection (ensureConnection, reconnect)
- Folder path normalization with delimiter detection
- Gmail-specific logic in ensureFolder (uses "/" delimiter, handles system folders)
- Email content extraction handles multipart messages and quoted-printable encoding
- Draft formatting includes HTML response + quoted original message
- Panic recovery in SaveDraft and MarkAsProcessed
**AI Integration** (internal/ai/ai.go):
- Two-stage process: language detection, then reply generation
- Language-aware responses (system prompt includes detected language)
- Output must be raw HTML (not markdown-wrapped)
- Retry logic with max 3 attempts
**Context Fetching** (internal/fetcher/fetcher.go):
- HTML to plain text extraction (strips tags, extracts text nodes)
- 30-second timeout per URL
- Batch fetching fails if any URL fails
### Important Implementation Details
**MIME Email Parsing** (internal/imap/imap.go:287-510):
- `extractMessageContent` handles multipart and single-part messages
- Content-Transfer-Encoding support: quoted-printable
- `cleanMessageContent` has a `performHeaderStripping` flag (true for fallback, false for parsed content)
- Extensive debug logging for troubleshooting email parsing issues
**Folder Management** (internal/imap/imap.go:512-663):
- `ensureFolder` detects Gmail vs generic IMAP servers
- Gmail: Uses "/" delimiter, skips CREATE for system folders like "[Gmail]/Drafts"
- Generic IMAP: Lists mailboxes to detect delimiter, creates folders if missing
- Has timeout protection (30s for LIST, 35s overall)
- Includes panic recovery
**Logging Strategy**:
- All modules use `internal/logger` wrapper around logrus
- Structured logging with fields (logrus.Fields)
- Debug level logs include detailed information (e.g., email content lengths, folder paths)
- Recent commits show enhanced logging for troubleshooting email extraction
## Development Guidelines
### Error Handling
- Use defer/recover for panic protection in critical IMAP operations (SaveDraft, MarkAsProcessed, ensureFolder)
- Retry mechanisms in AI client (max 3 attempts)
- IMAP connection auto-reconnection on failure
- Continue processing remaining emails if one fails
### Adding New Features
- **New AI models**: Update config.yaml model parameter (passed to OpenRouter API)
- **New context sources**: Add URLs to config.yaml context.urls array
- **New IMAP operations**: Add methods to IMAPClient struct, ensure connection handling
- **Email format changes**: Modify draft template in SaveDraft (line 242-266)
### Testing Considerations
- Email parsing logic is complex (multipart, encoding) - test with various email formats
- Gmail behavior differs from generic IMAP servers
- Test with different folder delimiters ("/" vs ".")
- Language detection affects response language
## Dependencies
Core dependencies (go.mod):
- `github.com/emersion/go-imap` - IMAP client library
- `github.com/sirupsen/logrus` - Structured logging
- `golang.org/x/net/html` - HTML parsing for context extraction
- `gopkg.in/yaml.v3` - Configuration file parsing
## Common Issues
**IMAP Folder Issues**:
- Check logs for delimiter detection (ensureFolder)
- Gmail requires special handling for system folders
- Folder names are case-sensitive
**Email Content Extraction**:
- Multi-part emails may require different handling
- Check Content-Transfer-Encoding header
- Recent fixes improved header stripping and content cleaning
**AI Response Issues**:
- Verify API key is correctly loaded from environment/file
- Check model name is valid for OpenRouter
- Response must be raw HTML (system prompt enforces this)

83
cmd/paraclub-ai-mailer/main.go
View File

@@ -12,7 +12,6 @@ import (
"paraclub-ai-mailer/internal/fetcher" "paraclub-ai-mailer/internal/fetcher"
"paraclub-ai-mailer/internal/imap" "paraclub-ai-mailer/internal/imap"
"paraclub-ai-mailer/internal/logger" "paraclub-ai-mailer/internal/logger"
"paraclub-ai-mailer/internal/tokens"
"github.com/sirupsen/logrus" "github.com/sirupsen/logrus"
) )
@@ -57,12 +56,6 @@ func main() {
fetcher := fetcher.New() fetcher := fetcher.New()
aiProcessor := ai.New(cfg.AI) aiProcessor := ai.New(cfg.AI)
tokenCounter, err := tokens.New()
if err != nil {
logger.WithError(err).Error("Failed to initialize token counter")
os.Exit(1)
}
logger.Debug("All components initialized successfully") logger.Debug("All components initialized successfully")
// Setup signal handling for graceful shutdown // Setup signal handling for graceful shutdown
@@ -83,7 +76,7 @@ func main() {
return return
case <-ticker.C: case <-ticker.C:
logger.Debug("Processing tick started") logger.Debug("Processing tick started")
processEmails(imapClient, fetcher, aiProcessor, tokenCounter, cfg) processEmails(imapClient, fetcher, aiProcessor, cfg)
logger.Debug("Processing tick completed") logger.Debug("Processing tick completed")
} }
} }
@@ -96,7 +89,7 @@ func main() {
logger.Info("Application shutdown complete") logger.Info("Application shutdown complete")
} }
func processEmails(imapClient *imap.IMAPClient, fetcher *fetcher.Fetcher, aiProcessor *ai.AI, tokenCounter *tokens.TokenCounter, cfg *config.Config) { func processEmails(imapClient *imap.IMAPClient, fetcher *fetcher.Fetcher, aiProcessor *ai.AI, cfg *config.Config) {
logger.Debug("Starting email processing cycle") logger.Debug("Starting email processing cycle")
// Fetch unprocessed emails // Fetch unprocessed emails
@@ -123,7 +116,7 @@ func processEmails(imapClient *imap.IMAPClient, fetcher *fetcher.Fetcher, aiProc
logger.WithField("contextCount", len(contextContent)).Debug("Successfully fetched context content") logger.WithField("contextCount", len(contextContent)).Debug("Successfully fetched context content")
// Process each email // Process each email
var processedCount, errorCount, skippedCount int var processedCount, errorCount int
for _, email := range emails { for _, email := range emails {
logger.WithFields(logrus.Fields{ logger.WithFields(logrus.Fields{
"subject": email.Subject, "subject": email.Subject,
@@ -131,83 +124,18 @@ func processEmails(imapClient *imap.IMAPClient, fetcher *fetcher.Fetcher, aiProc
"messageId": email.ID, "messageId": email.ID,
}).Info("Processing email") }).Info("Processing email")
// Extract clean email content (removes attachments, MIME boundaries, headers, converts HTML to text) // Generate AI response
cleanEmailContent := imap.ExtractMessageContent(email.Body) response, err := aiProcessor.GenerateReply(email.Body, contextContent)
logger.WithFields(logrus.Fields{
"subject": email.Subject,
"cleanSize": len(cleanEmailContent),
}).Debug("Extracted clean email content")
// Calculate token count for validation (use English as default to avoid API call)
// Language detection will happen during actual GenerateReply
systemPrompt := aiProcessor.BuildSystemPrompt("English")
userPrompt := aiProcessor.BuildUserPrompt(contextContent, cleanEmailContent)
estimatedTokens := tokenCounter.EstimateFullRequest(systemPrompt, userPrompt)
logger.WithFields(logrus.Fields{
"subject": email.Subject,
"estimatedTokens": estimatedTokens,
"maxTokens": cfg.Processing.MaxTokens,
}).Debug("Calculated token estimate for email")
// Check token limit
if cfg.Processing.MaxTokens > 0 && estimatedTokens > cfg.Processing.MaxTokens {
logger.WithFields(logrus.Fields{
"subject": email.Subject,
"from": email.From,
"estimatedTokens": estimatedTokens,
"maxTokens": cfg.Processing.MaxTokens,
}).Warn("Email exceeds token limit, marking as AI-processed but keeping in inbox")
skippedCount++
// Mark as AI-processed to prevent reprocessing, but DON'T move the email
if markErr := imapClient.MarkAsAIProcessed(email); markErr != nil {
logger.WithFields(logrus.Fields{
"subject": email.Subject,
"error": markErr,
}).Error("Failed to mark oversized email as AI-processed")
} else {
logger.WithField("subject", email.Subject).Info("Marked oversized email as AI-processed (email remains in inbox)")
}
continue
}
// Generate AI response with clean content
response, err := aiProcessor.GenerateReply(cleanEmailContent, contextContent)
if err != nil { if err != nil {
logger.WithFields(logrus.Fields{ logger.WithFields(logrus.Fields{
"subject": email.Subject, "subject": email.Subject,
"error": err, "error": err,
}).Error("Failed to generate reply") }).Error("Failed to generate reply")
errorCount++ errorCount++
// Mark as AI-processed even on failure to prevent infinite retry loop
if markErr := imapClient.MarkAsAIProcessed(email); markErr != nil {
logger.WithFields(logrus.Fields{
"subject": email.Subject,
"error": markErr,
}).Error("Failed to mark failed email as AI-processed")
} else {
logger.WithField("subject", email.Subject).Info("Marked failed email as AI-processed to prevent reprocessing")
}
continue continue
} }
logger.WithField("responseLength", len(response)).Debug("Generated AI response") logger.WithField("responseLength", len(response)).Debug("Generated AI response")
// Mark as AI-processed immediately to prevent reprocessing if subsequent steps fail
if err := imapClient.MarkAsAIProcessed(email); err != nil {
logger.WithFields(logrus.Fields{
"subject": email.Subject,
"error": err,
}).Warn("Failed to mark email as AI-processed, continuing with draft save")
// Continue anyway - if this fails, worst case is we might reprocess
// But we want to try to save the draft we just generated
} else {
logger.Debug("Marked email as AI-processed")
}
// Save as draft // Save as draft
if err := imapClient.SaveDraft(email, response); err != nil { if err := imapClient.SaveDraft(email, response); err != nil {
logger.WithFields(logrus.Fields{ logger.WithFields(logrus.Fields{
@@ -240,6 +168,5 @@ func processEmails(imapClient *imap.IMAPClient, fetcher *fetcher.Fetcher, aiProc
"totalEmails": len(emails), "totalEmails": len(emails),
"processed": processedCount, "processed": processedCount,
"errors": errorCount, "errors": errorCount,
"skipped": skippedCount,
}).Info("Completed email processing cycle") }).Info("Completed email processing cycle")
} }

7
config.yaml.example
View File

@@ -23,13 +23,6 @@ context:
polling: polling:
interval: "5m" # Examples: "30s", "1m", "1h" interval: "5m" # Examples: "30s", "1m", "1h"
processing:
max_tokens: 12000 # Maximum total tokens for API request (system + context + email)
# Recommended: 8000-12000 for most models
# Set to 0 for no limit
# Note: Attachments are automatically stripped before counting
skip_junk_emails: false # Skip emails marked as junk/spam (not yet implemented)
logging: logging:
level: "info" # Options: "debug", "info", "warn", "error" level: "info" # Options: "debug", "info", "warn", "error"
file_path: "paraclub-ai-mailer.log" file_path: "paraclub-ai-mailer.log"

18
config/config.go
View File

@@ -11,12 +11,11 @@ import (
) )
type Config struct { type Config struct {
IMAP IMAPConfig `yaml:"imap"` IMAP IMAPConfig `yaml:"imap"`
AI AIConfig `yaml:"ai"` AI AIConfig `yaml:"ai"`
Context ContextConfig `yaml:"context"` Context ContextConfig `yaml:"context"`
Polling PollingConfig `yaml:"polling"` Polling PollingConfig `yaml:"polling"`
Logging LoggingConfig `yaml:"logging"` Logging LoggingConfig `yaml:"logging"`
Processing ProcessingConfig `yaml:"processing"`
} }
type IMAPConfig struct { type IMAPConfig struct {
@@ -50,11 +49,6 @@ type LoggingConfig struct {
FilePath string `yaml:"file_path"` FilePath string `yaml:"file_path"`
} }
type ProcessingConfig struct {
MaxTokens int `yaml:"max_tokens"` // Maximum total tokens for API request (0 = no limit)
SkipJunkEmails bool `yaml:"skip_junk_emails"` // Skip emails marked as junk/spam
}
func readFileContent(path string) (string, error) { func readFileContent(path string) (string, error) {
content, err := os.ReadFile(path) content, err := os.ReadFile(path)
if err != nil { if err != nil {
@@ -125,8 +119,6 @@ func Load(path string) (*Config, error) {
"pollingInterval": config.Polling.Interval, "pollingInterval": config.Polling.Interval,
"loggingLevel": config.Logging.Level, "loggingLevel": config.Logging.Level,
"loggingFilePath": config.Logging.FilePath, "loggingFilePath": config.Logging.FilePath,
"maxTokens": config.Processing.MaxTokens,
"skipJunkEmails": config.Processing.SkipJunkEmails,
}).Debug("Configuration loaded successfully") }).Debug("Configuration loaded successfully")
return &config, nil return &config, nil

3
go.mod
View File

@@ -10,10 +10,7 @@ require (
) )
require ( require (
github.com/dlclark/regexp2 v1.10.0 // indirect
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 // indirect github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/pkoukk/tiktoken-go v0.1.8 // indirect
golang.org/x/sys v0.30.0 // indirect golang.org/x/sys v0.30.0 // indirect
golang.org/x/text v0.22.0 // indirect golang.org/x/text v0.22.0 // indirect
) )

7
go.sum
View File

@@ -1,8 +1,6 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dlclark/regexp2 v1.10.0 h1:+/GIL799phkJqYW+3YbOd8LCcbHzT0Pbo8zl70MHsq0=
github.com/dlclark/regexp2 v1.10.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/emersion/go-imap v1.2.1 h1:+s9ZjMEjOB8NzZMVTM3cCenz2JrQIGGo5j1df19WjTA= github.com/emersion/go-imap v1.2.1 h1:+s9ZjMEjOB8NzZMVTM3cCenz2JrQIGGo5j1df19WjTA=
github.com/emersion/go-imap v1.2.1/go.mod h1:Qlx1FSx2FTxjnjWpIlVNEuX+ylerZQNFE5NsmKFSejY= github.com/emersion/go-imap v1.2.1/go.mod h1:Qlx1FSx2FTxjnjWpIlVNEuX+ylerZQNFE5NsmKFSejY=
github.com/emersion/go-message v0.15.0/go.mod h1:wQUEfE+38+7EW8p8aZ96ptg6bAb1iwdgej19uXASlE4= github.com/emersion/go-message v0.15.0/go.mod h1:wQUEfE+38+7EW8p8aZ96ptg6bAb1iwdgej19uXASlE4=
@@ -10,10 +8,6 @@ github.com/emersion/go-sasl v0.0.0-20200509203442-7bfe0ed36a21/go.mod h1:iL2twTe
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 h1:oP4q0fw+fOSWn3DfFi4EXdT+B+gTtzx8GC9xsc26Znk= github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 h1:oP4q0fw+fOSWn3DfFi4EXdT+B+gTtzx8GC9xsc26Znk=
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6/go.mod h1:iL2twTeMvZnrg54ZoPDNfJaJaqy0xIQFuBdrLsmspwQ= github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6/go.mod h1:iL2twTeMvZnrg54ZoPDNfJaJaqy0xIQFuBdrLsmspwQ=
github.com/emersion/go-textwrapper v0.0.0-20200911093747-65d896831594/go.mod h1:aqO8z8wPrjkscevZJFVE1wXJrLpC5LtJG7fqLOsPb2U= github.com/emersion/go-textwrapper v0.0.0-20200911093747-65d896831594/go.mod h1:aqO8z8wPrjkscevZJFVE1wXJrLpC5LtJG7fqLOsPb2U=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/pkoukk/tiktoken-go v0.1.8 h1:85ENo+3FpWgAACBaEUVp+lctuTcYUO7BtmfhlN/QTRo=
github.com/pkoukk/tiktoken-go v0.1.8/go.mod h1:9NiV+i9mJKGj1rYOT+njbv+ZwA/zJxYdewGl6qVatpg=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
@@ -21,7 +15,6 @@ github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVs
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY= github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8=
golang.org/x/net v0.35.0 h1:T5GQRQb2y08kTAByq9L4/bz8cipCdA8FbRTXewonqY8= golang.org/x/net v0.35.0 h1:T5GQRQb2y08kTAByq9L4/bz8cipCdA8FbRTXewonqY8=
golang.org/x/net v0.35.0/go.mod h1:EglIi67kWsHKlRzzVMUD93VMSWGFOMSZgxFjparz1Qk= golang.org/x/net v0.35.0/go.mod h1:EglIi67kWsHKlRzzVMUD93VMSWGFOMSZgxFjparz1Qk=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=

61
internal/ai/ai.go
View File

@@ -43,31 +43,6 @@ func New(cfg config.AIConfig) *AI {
} }
} }
// BuildSystemPrompt creates the system prompt for a given language
// Exposed for token counting without making an API call
func (a *AI) BuildSystemPrompt(lang string) string {
return fmt.Sprintf(`You are a helpful assistant who responds to emails.
Your primary goal is to answer the user's query (found in the 'Email Body') by primarily using the information available in the 'Additional Context' and your general knowledge.
While the 'Email Body' provides the question, your answer should be synthesized from the context and your understanding, not by directly repeating or solely relying on unverified information from the 'Email Body' itself.
Instructions:
- Language: Your response must be entirely in %s, regardless of the language used in the email content or context.
- Format: CRITICAL: Your reply MUST be raw HTML. Use appropriate HTML tags for structure and styling. For example, wrap paragraphs in <p>...</p> tags and use <br> for line breaks if needed within a paragraph. Even a short sentence must be wrapped in HTML (e.g., <p>Yes.</p>).
- Markdown: Do NOT wrap the HTML in markdown code blocks (e.g., %s).
- Extraneous Text: Do not include a subject line. Do not include explanations, commentary, or any extra text that is not part of the direct answer.
- Closing: Avoid generic closing statements like "If you have further questions...". Focus solely on answering the email.
`, lang, "```html ... ```")
}
// BuildUserPrompt creates the user message with context and email content
func (a *AI) BuildUserPrompt(contextContent map[string]string, emailContent string) string {
var context string
for url, content := range contextContent {
context += fmt.Sprintf("\nContext from %s:\n%s\n", url, content)
}
return fmt.Sprintf("### Additional Context:\n%s\n\n### Email Body:\n%s", context, emailContent)
}
func (a *AI) detectLanguage(emailContent string) (string, error) { func (a *AI) detectLanguage(emailContent string) (string, error) {
logger.WithField("emailContentLength", len(emailContent)).Debug("Starting language detection") logger.WithField("emailContentLength", len(emailContent)).Debug("Starting language detection")
@@ -102,24 +77,38 @@ func (a *AI) GenerateReply(emailContent string, contextContent map[string]string
return "", err return "", err
} }
// Build prompts using exposed methods // Prepare context from all URLs
systemMsg := a.BuildSystemPrompt(lang) var context string
userMsg := a.BuildUserPrompt(contextContent, emailContent) for url, content := range contextContent {
context += fmt.Sprintf("\nContext from %s:\n%s\n", url, content)
logger.WithFields(logrus.Fields{
"url": url,
"contentLength": len(content),
}).Debug("Added context from URL")
}
logger.Debug("Generated system and user prompts") // Prepare the system message with language-specific instruction
systemMsg := fmt.Sprintf(`You are a helpful assistant who responds to emails.
Your primary goal is to answer the user's query based on the provided email and context.
Instructions:
- Language: Your response must be entirely in %s, regardless of the language used in the email content or context.
- Format: Your reply must be raw HTML. Use appropriate HTML tags for structure and styling.
- Markdown: Do NOT wrap the HTML in markdown code blocks (e.g., %s).
- Extraneous Text: Do not include a subject line. Do not include explanations, commentary, or any extra text that is not part of the direct answer.
- Closing: Avoid generic closing statements like "If you have further questions...". Focus solely on answering the email.
`, lang, "```html ... ```")
logger.WithFields(logrus.Fields{
"systemprompt": systemMsg,
}).Debug("Generating system prompt")
userMsg := fmt.Sprintf("### Additional Context:\n%s\n\n### Email Body:\n%s", context, emailContent)
messages := []Message{ messages := []Message{
{Role: "system", Content: systemMsg}, {Role: "system", Content: systemMsg},
{Role: "user", Content: userMsg}, {Role: "user", Content: userMsg},
} }
aiResponse, err := a.makeAPIRequest(messages) return a.makeAPIRequest(messages)
if err != nil {
// Error already logged by makeAPIRequest, just propagate
return "", err
}
logger.WithField("rawAIResponse", aiResponse).Debug("Received raw response from AI")
return aiResponse, nil
} }
func (a *AI) makeAPIRequest(messages []Message) (string, error) { func (a *AI) makeAPIRequest(messages []Message) (string, error) {

319
internal/imap/imap.go
View File

@@ -6,7 +6,6 @@ import (
"io" "io"
"mime" "mime"
"mime/multipart" "mime/multipart"
"mime/quotedprintable"
"net/mail" "net/mail"
"paraclub-ai-mailer/config" "paraclub-ai-mailer/config"
"paraclub-ai-mailer/internal/logger" "paraclub-ai-mailer/internal/logger"
@@ -18,9 +17,6 @@ import (
"github.com/sirupsen/logrus" "github.com/sirupsen/logrus"
) )
// Custom IMAP flag to mark emails as AI-processed
const AIProcessedFlag = "$AIProcessed"
type IMAPClient struct { type IMAPClient struct {
client *client.Client client *client.Client
config config.IMAPConfig config config.IMAPConfig
@@ -77,7 +73,6 @@ type Email struct {
ID string ID string
Subject string Subject string
From string From string
Date time.Time
Body string Body string
} }
@@ -140,9 +135,9 @@ func (ic *IMAPClient) FetchUnprocessedEmails() ([]Email, error) {
return nil, nil return nil, nil
} }
// Get all messages in the inbox that haven't been seen yet and haven't been AI-processed // Get all messages in the inbox that haven't been seen yet
criteria := imap.NewSearchCriteria() criteria := imap.NewSearchCriteria()
criteria.WithoutFlags = []string{"\\Seen", AIProcessedFlag} criteria.WithoutFlags = []string{"\\Seen"}
uids, err := ic.client.Search(criteria) uids, err := ic.client.Search(criteria)
if err != nil { if err != nil {
@@ -199,7 +194,6 @@ func (ic *IMAPClient) FetchUnprocessedEmails() ([]Email, error) {
ID: msg.Envelope.MessageId, ID: msg.Envelope.MessageId,
Subject: msg.Envelope.Subject, Subject: msg.Envelope.Subject,
From: from, From: from,
Date: msg.Envelope.Date,
Body: bodyBuilder.String(), Body: bodyBuilder.String(),
}) })
} }
@@ -265,8 +259,8 @@ func (ic *IMAPClient) SaveDraft(email Email, response string) (err error) {
response, response,
email.From, email.From,
email.Subject, email.Subject,
email.Date.Format("Mon, 02 Jan 2006 15:04:05 -0700"), time.Now().Format("Mon, 02 Jan 2006 15:04:05 -0700"),
ExtractMessageContent(email.Body)) extractMessageContent(email.Body))
literal := &MessageLiteral{ literal := &MessageLiteral{
content: []byte(draft), content: []byte(draft),
@@ -287,24 +281,21 @@ func (ic *IMAPClient) SaveDraft(email Email, response string) (err error) {
return nil return nil
} }
// ExtractMessageContent attempts to extract just the message content // extractMessageContent attempts to extract just the message content
// by removing email headers and MIME boundaries // by removing email headers and MIME boundaries
func ExtractMessageContent(body string) string { func extractMessageContent(body string) string {
logger.WithField("bodyLength", len(body)).Debug("Starting message content extraction") logger.WithField("bodyLength", len(body)).Debug("Starting message content extraction")
logger.WithField("rawInputBody", body).Debug("extractMessageContent: Raw input body")
msg, err := mail.ReadMessage(strings.NewReader(body)) msg, err := mail.ReadMessage(strings.NewReader(body))
if err != nil { if err != nil {
logger.WithFields(logrus.Fields{ logger.WithFields(logrus.Fields{
"error": err, "error": err,
"bodyLength": len(body), "bodyLength": len(body),
}).Debug("Failed to parse email message, falling back to simple extraction") }).Debug("Failed to parse email message, falling back to simple extraction")
// When ReadMessage fails, the body is raw, so header stripping is needed. return fallbackExtractContent(body)
return cleanMessageContent(fallbackExtractContent(body), true)
} }
contentTypeHeader := msg.Header.Get("Content-Type") contentTypeHeader := msg.Header.Get("Content-Type")
logger.WithField("contentTypeHeader", contentTypeHeader).Debug("Got Content-Type header") logger.WithField("contentTypeHeader", contentTypeHeader).Debug("Got Content-Type header")
logger.WithField("parsedContentTypeHeader", contentTypeHeader).Debug("extractMessageContent: Parsed Content-Type header")
mediaType, params, err := mime.ParseMediaType(contentTypeHeader) mediaType, params, err := mime.ParseMediaType(contentTypeHeader)
if err != nil { if err != nil {
@@ -312,69 +303,41 @@ func ExtractMessageContent(body string) string {
"error": err, "error": err,
"contentTypeHeader": contentTypeHeader, "contentTypeHeader": contentTypeHeader,
}).Debug("Failed to parse Content-Type header, falling back to simple extraction") }).Debug("Failed to parse Content-Type header, falling back to simple extraction")
// When ParseMediaType fails, the body is raw, so header stripping is needed. return fallbackExtractContent(body)
return cleanMessageContent(fallbackExtractContent(body), true)
} }
logger.WithFields(logrus.Fields{ logger.WithFields(logrus.Fields{
"mediaType": mediaType, "mediaType": mediaType,
"params": params, "params": params,
}).Debug("Parsed message Content-Type") }).Debug("Parsed message Content-Type")
logger.WithFields(logrus.Fields{
"mediaType": mediaType,
"params": params,
}).Debug("extractMessageContent: Parsed mediaType and params")
var content string var content string
if strings.HasPrefix(mediaType, "multipart/") { if strings.HasPrefix(mediaType, "multipart/") {
// For multipart, the handling of Content-Transfer-Encoding will be done within handleMultipartMessage for each part
logger.Debug("extractMessageContent: Handling as multipart message")
content = handleMultipartMessage(msg.Body, params["boundary"]) content = handleMultipartMessage(msg.Body, params["boundary"])
logger.WithField("contentFromMultipart", content).Debug("extractMessageContent: Content after handleMultipartMessage")
} else { } else {
// For single part, handle Content-Transfer-Encoding here content = handleSinglePartMessage(msg.Body)
var partReader io.Reader = msg.Body
transferEncoding := strings.ToLower(msg.Header.Get("Content-Transfer-Encoding"))
if transferEncoding == "quoted-printable" {
partReader = quotedprintable.NewReader(msg.Body)
}
// Add handling for "base64" if needed in the future
// else if transferEncoding == "base64" {
// partReader = base64.NewDecoder(base64.StdEncoding, msg.Body)
// }
logger.Debug("extractMessageContent: Handling as single part message")
content = handleSinglePartMessage(partReader)
logger.WithField("contentFromSinglePart", content).Debug("extractMessageContent: Content after handleSinglePartMessage")
} }
if content == "" { if content == "" {
logger.Debug("No content extracted, falling back to simple extraction") logger.Debug("No content extracted, falling back to simple extraction")
logger.Debug("extractMessageContent: No content from primary extraction, falling back.") return fallbackExtractContent(body)
// When primary extraction yields no content, the body is raw, so header stripping is needed.
return cleanMessageContent(fallbackExtractContent(body), true)
} }
// Clean up the content // Clean up the content
logger.WithField("contentBeforeClean", content).Debug("extractMessageContent: Content before cleanMessageContent") content = cleanMessageContent(content)
// When ReadMessage succeeds, 'content' is already the message body (or part body),
// so no further header stripping should be done.
content = cleanMessageContent(content, false)
logger.WithField("contentAfterClean", content).Debug("extractMessageContent: Content after cleanMessageContent")
logger.WithField("contentLength", len(content)).Debug("Successfully extracted and cleaned message content") logger.WithField("contentLength", len(content)).Debug("Successfully extracted and cleaned message content")
return content return content
} }
func handleMultipartMessage(reader io.Reader, boundary string) string { func handleMultipartMessage(reader io.Reader, boundary string) string {
logger.WithField("boundary", boundary).Debug("handleMultipartMessage: Starting with boundary")
if boundary == "" { if boundary == "" {
logger.Debug("No boundary found in multipart message") logger.Debug("No boundary found in multipart message")
return "" return ""
} }
mReader := multipart.NewReader(reader, boundary) mReader := multipart.NewReader(reader, boundary)
var textPlainContent string var textContent string
var textHTMLContent string
partIndex := 0 partIndex := 0
for { for {
@@ -388,206 +351,89 @@ func handleMultipartMessage(reader io.Reader, boundary string) string {
} }
contentType := part.Header.Get("Content-Type") contentType := part.Header.Get("Content-Type")
contentDisposition := part.Header.Get("Content-Disposition")
contentTransferEncoding := strings.ToLower(part.Header.Get("Content-Transfer-Encoding"))
logger.WithFields(logrus.Fields{
"partIndex": partIndex,
"partContentType": contentType,
"partDisposition": contentDisposition,
"partTransferEncoding": contentTransferEncoding,
}).Debug("handleMultipartMessage: Processing part")
// Skip attachments
if strings.HasPrefix(contentDisposition, "attachment") {
logger.WithFields(logrus.Fields{
"partIndex": partIndex,
"filename": part.FileName(),
}).Debug("Skipping attachment part")
partIndex++
continue
}
// Skip non-text content types (images, videos, applications, etc.)
if !strings.HasPrefix(contentType, "text/plain") &&
!strings.HasPrefix(contentType, "text/html") {
logger.WithFields(logrus.Fields{
"partIndex": partIndex,
"contentType": contentType,
}).Debug("Skipping non-text content type")
partIndex++
continue
}
// Handle quoted-printable encoding
var partReader io.Reader = part
if contentTransferEncoding == "quoted-printable" {
partReader = quotedprintable.NewReader(part)
}
buf := new(bytes.Buffer)
if _, err := buf.ReadFrom(partReader); err != nil {
logger.WithError(err).WithField("partIndex", partIndex).Debug("Failed to read from partReader in multipart")
partIndex++
continue
}
// Store text/plain and text/html separately
if strings.HasPrefix(contentType, "text/plain") { if strings.HasPrefix(contentType, "text/plain") {
textPlainContent = buf.String() buf := new(bytes.Buffer)
logger.WithField("textPlainLength", len(textPlainContent)).Debug("Found text/plain part") if _, err := buf.ReadFrom(part); err != nil {
} else if strings.HasPrefix(contentType, "text/html") { continue
textHTMLContent = buf.String() }
logger.WithField("textHTMLLength", len(textHTMLContent)).Debug("Found text/html part") textContent = buf.String()
break
} }
partIndex++ partIndex++
} }
// Prefer text/plain over text/html return textContent
if textPlainContent != "" {
logger.Debug("handleMultipartMessage: Returning text/plain content")
return textPlainContent
}
if textHTMLContent != "" {
logger.Debug("handleMultipartMessage: Converting text/html to plain text")
return htmlToPlainText(textHTMLContent)
}
logger.Debug("handleMultipartMessage: No text content found")
return ""
} }
func handleSinglePartMessage(reader io.Reader) string { func handleSinglePartMessage(reader io.Reader) string {
logger.Debug("handleSinglePartMessage: Starting")
buf := new(bytes.Buffer) buf := new(bytes.Buffer)
if _, err := buf.ReadFrom(reader); err != nil { if _, err := buf.ReadFrom(reader); err != nil {
logger.WithError(err).Debug("Failed to read message body") logger.WithError(err).Debug("Failed to read message body")
return "" return ""
} }
content := buf.String() return buf.String()
logger.WithField("readContent", content).Debug("handleSinglePartMessage: Content read from reader")
return content
} }
// htmlToPlainText converts HTML content to plain text by extracting text nodes func cleanMessageContent(content string) string {
func htmlToPlainText(htmlContent string) string { // Remove any remaining email headers that might be in the body
logger.WithField("htmlLength", len(htmlContent)).Debug("Converting HTML to plain text") lines := strings.Split(content, "\n")
// Simple HTML tag stripping - removes all HTML tags and extracts text
var result strings.Builder
inTag := false
for _, char := range htmlContent {
switch char {
case '<':
inTag = true
case '>':
inTag = false
default:
if !inTag {
result.WriteRune(char)
}
}
}
plainText := result.String()
// Clean up excessive whitespace
plainText = strings.ReplaceAll(plainText, "\r\n", "\n")
plainText = strings.ReplaceAll(plainText, "\r", "\n")
// Replace multiple spaces with single space
lines := strings.Split(plainText, "\n")
var cleanLines []string var cleanLines []string
headerSection := true
for _, line := range lines { for _, line := range lines {
trimmed := strings.TrimSpace(line) trimmed := strings.TrimSpace(line)
if trimmed != "" {
cleanLines = append(cleanLines, trimmed) // Empty line marks the end of headers
if headerSection && trimmed == "" {
headerSection = false
continue
} }
}
result2 := strings.Join(cleanLines, "\n") // Skip header lines
logger.WithField("plainTextLength", len(result2)).Debug("HTML converted to plain text") if headerSection && (strings.Contains(trimmed, ":") || trimmed == "") {
return result2 continue
} }
func cleanMessageContent(content string, performHeaderStripping bool) string { // Add non-header lines
logger.WithField("inputContentLength", len(content)).Debug("cleanMessageContent: Starting") if !headerSection {
logger.WithField("performHeaderStripping", performHeaderStripping).Debug("cleanMessageContent: performHeaderStripping flag")
if performHeaderStripping {
// Remove any remaining email headers that might be in the body
lines := strings.Split(content, "\n")
var cleanLines []string
headerSection := true
for _, line := range lines {
trimmed := strings.TrimSpace(line)
// Empty line marks the end of headers
if headerSection && trimmed == "" {
headerSection = false
continue
}
// Skip header lines
if headerSection && (strings.Contains(trimmed, ":") || trimmed == "") {
continue
}
// Add non-header lines
// This condition was originally !headerSection, but if we are past the headers,
// we should always add the line. If headerSection is still true here, it means
// it's the first line of content after potential headers were skipped.
cleanLines = append(cleanLines, line) cleanLines = append(cleanLines, line)
} }
content = strings.Join(cleanLines, "\n")
logger.WithField("contentAfterHeaderStripLength", len(content)).Debug("cleanMessageContent: Content after header stripping")
} else {
logger.Debug("cleanMessageContent: Skipping header stripping")
} }
content = strings.Join(cleanLines, "\n")
// Convert newlines to HTML breaks for display // Convert newlines to HTML breaks for display
// First, normalize all newlines (\r\n or \n) to just \n content = strings.ReplaceAll(content, "\r\n", "<br>\n")
content = strings.ReplaceAll(content, "\r\n", "\n")
// Then, replace each \n with <br>\n
content = strings.ReplaceAll(content, "\n", "<br>\n") content = strings.ReplaceAll(content, "\n", "<br>\n")
logger.WithField("contentAfterNewlineConversionLength", len(content)).Debug("cleanMessageContent: Content after newline conversion")
// Remove any remaining email signature markers // Remove any remaining email signature markers
content = strings.Split(content, "\n-- ")[0] content = strings.Split(content, "\n-- ")[0]
content = strings.Split(content, "<br>-- ")[0] content = strings.Split(content, "<br>-- ")[0]
finalContent := strings.TrimSpace(content) return strings.TrimSpace(content)
logger.WithField("finalCleanedContentLength", len(finalContent)).Debug("cleanMessageContent: Returning final cleaned content")
return finalContent
} }
// fallbackExtractContent is the previous implementation used as fallback // fallbackExtractContent is the previous implementation used as fallback
func fallbackExtractContent(body string) string { func fallbackExtractContent(body string) string {
logger.WithField("bodyLength", len(body)).Debug("Using fallback content extraction method") logger.WithField("bodyLength", len(body)).Debug("Using fallback content extraction method")
logger.WithField("rawInputBodyFallbackLength", len(body)).Debug("fallbackExtractContent: Raw input body")
var content string
parts := strings.Split(body, "\r\n\r\n") parts := strings.Split(body, "\r\n\r\n")
if len(parts) > 1 { if len(parts) > 1 {
content = strings.Join(parts[1:], "\r\n\r\n") content := strings.Join(parts[1:], "\r\n\r\n")
content = strings.ReplaceAll(content, "\r\n", "<br>\n")
content = strings.ReplaceAll(content, "\n", "<br>\n")
logger.WithFields(logrus.Fields{ logger.WithFields(logrus.Fields{
"contentLength": len(content), "contentLength": len(content),
"partsCount": len(parts), "partsCount": len(parts),
}).Debug("Successfully extracted content using fallback method (from parts)") }).Debug("Successfully extracted content using fallback method")
logger.WithField("extractedContentFallbackLength", len(content)).Debug("fallbackExtractContent: Content from splitting parts") return content
} else {
content = body
logger.WithFields(logrus.Fields{
"contentLength": len(content),
"fullBody": true,
}).Debug("Using full body as content in fallback method")
logger.WithField("finalContentFallbackLength", len(content)).Debug("fallbackExtractContent: Final content from full body (no parts split)")
} }
// Newline conversion and signature stripping will be handled by cleanMessageContent, content := body
// which is now called by the callers of fallbackExtractContent. content = strings.ReplaceAll(content, "\r\n", "<br>\n")
content = strings.ReplaceAll(content, "\n", "<br>\n")
logger.WithFields(logrus.Fields{
"contentLength": len(content),
"fullBody": true,
}).Debug("Using full body as content in fallback method")
return content return content
} }
@@ -820,67 +666,6 @@ func (ic *IMAPClient) MarkAsProcessed(email Email) (err error) {
return nil return nil
} }
// MarkAsAIProcessed marks an email with a custom flag to indicate AI has processed it
// This prevents reprocessing the same email if subsequent operations (SaveDraft, MarkAsProcessed) fail
func (ic *IMAPClient) MarkAsAIProcessed(email Email) (err error) {
defer func() {
if r := recover(); r != nil {
logger.WithFields(logrus.Fields{
"panic": r,
"emailSubject": email.Subject,
}).Error("Panic occurred in MarkAsAIProcessed")
err = fmt.Errorf("panic in MarkAsAIProcessed: %v", r)
}
}()
if err := ic.ensureConnection(); err != nil {
return fmt.Errorf("failed to ensure IMAP connection: %v", err)
}
// Select source mailbox
_, err = ic.client.Select(ic.config.MailboxIn, false)
if err != nil {
return fmt.Errorf("failed to select source mailbox: %v", err)
}
// Find the email by Message-Id
criteria := imap.NewSearchCriteria()
criteria.Header.Set("Message-Id", email.ID)
uids, err := ic.client.Search(criteria)
if err != nil {
return fmt.Errorf("search failed: %v", err)
}
if len(uids) == 0 {
return fmt.Errorf("email not found")
}
// Mark with AI-processed flag
seqSet := new(imap.SeqSet)
seqSet.AddNum(uids...)
item := imap.FormatFlagsOp(imap.AddFlags, true)
flags := []interface{}{AIProcessedFlag}
if err := ic.client.Store(seqSet, item, flags, nil); err != nil {
// If custom flags are not supported, fall back to marking as \Seen
logger.WithFields(logrus.Fields{
"error": err,
"subject": email.Subject,
}).Warn("Failed to set custom AI-processed flag, falling back to \\Seen flag")
flags = []interface{}{imap.SeenFlag}
if err := ic.client.Store(seqSet, item, flags, nil); err != nil {
return fmt.Errorf("failed to mark message with fallback flag: %v", err)
}
logger.WithField("subject", email.Subject).Info("Marked email as \\Seen (custom flag not supported)")
return nil
}
logger.WithField("subject", email.Subject).Debug("Successfully marked email with AI-processed flag")
return nil
}
func (ic *IMAPClient) Close() error { func (ic *IMAPClient) Close() error {
if ic.client != nil { if ic.client != nil {
ic.client.Logout() ic.client.Logout()

50
internal/tokens/tokens.go
View File

@@ -1,50 +0,0 @@
package tokens
import (
"fmt"
"paraclub-ai-mailer/internal/logger"
"github.com/pkoukk/tiktoken-go"
"github.com/sirupsen/logrus"
)
type TokenCounter struct {
encoding *tiktoken.Tiktoken
}
// New creates a token counter with cl100k_base encoding (GPT-4/Claude compatible)
func New() (*TokenCounter, error) {
enc, err := tiktoken.GetEncoding("cl100k_base")
if err != nil {
return nil, fmt.Errorf("failed to get tiktoken encoding: %w", err)
}
return &TokenCounter{encoding: enc}, nil
}
// CountString counts tokens in a single string
func (tc *TokenCounter) CountString(text string) int {
tokens := tc.encoding.Encode(text, nil, nil)
return len(tokens)
}
// EstimateFullRequest estimates total tokens for the complete API request
// Includes: system prompt + user prompt + message overhead
func (tc *TokenCounter) EstimateFullRequest(systemPrompt, userPrompt string) int {
systemTokens := tc.CountString(systemPrompt)
userTokens := tc.CountString(userPrompt)
// Add overhead for message structure (~100 tokens for JSON formatting, role labels, etc.)
overhead := 100
total := systemTokens + userTokens + overhead
logger.WithFields(logrus.Fields{
"systemTokens": systemTokens,
"userTokens": userTokens,
"overhead": overhead,
"total": total,
}).Debug("Token estimation breakdown")
return total
}