Gmail Spam & Memory Leak Bugfix¶

Date: November 1, 2025 Status: FIXED ✅ Severity: CRITICAL (Production Incident)

Problem Summary¶

The Gmail Mind Reader workflow was sending repeated messages and caused the Render server to run out of memory, requiring manual shutdown.

Root Causes Identified¶

1. Proactive Scheduler Running Without User Knowledge ⚠️¶

Location: app/main.py:97-111, app/scheduler/background_service.py:87-112

The proactive email urgency workflow was running every 15 minutes (8am-8pm) for hardcoded user ID 1967236542, independent of manual user requests.

Impact: - User asked for emails once - Scheduler ALSO checked every 15 minutes - Both sent notifications → appeared as spam - Scheduler kept running until server shutdown

2. Memory Leak from Email Fetching 💾¶

Location: app/superpowers/catalog/proactive/email_urgency.py:42-46

The workflow fetched 50 emails with include_body: True, loading full email bodies (50KB+ each) into memory.

Memory Usage: - 50 emails × 50KB = 2.5+ MB per run - Every 15 min × 4 runs/hour = 10+ MB/hour - No cleanup between runs - Render free tier: 512MB RAM limit → OOM crash

3. Excessive LLM Token Usage 💸¶

Location: app/superpowers/catalog/proactive/email_urgency.py:169-220

Analyzed 20 emails per run
10 memories per context
200-char snippets
800 max_tokens for response
Long system prompt

Cost per run: ~2,000 tokens × 4 runs/hour × 12 hours = 96K tokens/day = ~$2-3/day per user

4. No OAuth Validation¶

Location: app/scheduler/background_service.py:187-191

The scheduler executed workflows even if users didn't have valid OAuth tokens, causing repeated auth failures without stopping.

5. No Circuit Breaker¶

Failed workflows retried infinitely without backoff, accumulating errors and memory.

Fixes Implemented¶

✅ Fix 1: Kill Switch (Scheduler Disabled by Default)¶

File: app/main.py:97-111

# Kill switch: only start if explicitly enabled
enable_scheduler = os.getenv("ENABLE_PROACTIVE_SCHEDULER", "false").lower() == "true"

if not enable_scheduler:
    logger.info("Proactive scheduler DISABLED")
    return

Default: Scheduler is OFF in production To enable: Set ENABLE_PROACTIVE_SCHEDULER=true in Render env vars

✅ Fix 2: Reduced Email Fetching (70% Memory Savings)¶

File: app/superpowers/catalog/proactive/email_urgency.py

Before: - 50 emails with full bodies - 20 memories - 200-char snippets

After: - 15 emails (reduced 70%) - include_body: False (use snippets only) - 5 memories (reduced 75%) - 150-char snippets (reduced 25%)

Memory savings: ~2MB → ~300KB per run (85% reduction)

✅ Fix 3: Reduced LLM Costs (80% Token Savings)¶

File: app/superpowers/catalog/proactive/email_urgency.py:169-196

Before: - 20 emails analyzed - 800 max_tokens - Long system prompt (~300 tokens)

After: - 10 emails (reduced 50%) - 300 max_tokens (reduced 62%) - Compact prompt (~80 tokens, 73% reduction)

Cost savings: 2,000 tokens → ~400 tokens per run (80% reduction) Daily cost: $2-3/day → **$0.40-0.60/day** per user

✅ Fix 4: OAuth Validation¶

File: app/scheduler/background_service.py:209-248

async def _get_active_users(self):
    """Get users who have OAuth connected and valid tokens"""
    query = text('''
        SELECT DISTINCT user_phone
        FROM oauth_tokens
        WHERE expires_at > NOW()
        AND scope IN ('gmail', 'both')
    ''')
    return users

async def _has_valid_gmail_token(self, user_phone: str) -> bool:
    """Check if user has valid Gmail OAuth token"""
    return await token_manager.has_valid_token(db, user_phone, "gmail")

Before: Hardcoded user ID, no token check After: Query database for users with valid, non-expired Gmail tokens

✅ Fix 5: Circuit Breaker with Exponential Backoff¶

File: app/scheduler/background_service.py:250-282

# Circuit breaker configuration
self.circuit_breaker_threshold = 3  # Stop after 3 failures

def _is_circuit_broken(self, workflow_id: str) -> bool:
    """Check if circuit breaker is open"""
    if failure_count >= 3:
        # Exponential backoff: 15min, 30min, 60min, max 180min
        backoff_minutes = 15 * (2 ** (failure_count - 3))
        backoff_minutes = min(backoff_minutes, 180)
        # Only retry after backoff period

Backoff schedule: - 3 failures → 15 min wait - 4 failures → 30 min wait - 5 failures → 60 min wait - 6+ failures → 180 min wait

Auto-reset: On success, circuit breaker resets to 0

Performance Impact¶

Memory Usage¶

Before: 2.5MB per run, 10MB/hour After: 300KB per run, 1.2MB/hour Savings: 85% reduction

LLM Costs (per user per day)¶

Before: ~96K tokens = $2-3/day After: ~19K tokens = $0.40-0.60/day Savings: 80% reduction

Server Stability¶

Before: OOM crashes, manual restarts required After: Stable, scheduler disabled by default

Testing Checklist¶

Scheduler disabled by default (verified in app/main.py)
Email fetch reduced to 15, no bodies (verified in workflow)
LLM costs reduced (300 tokens, compact prompt)
OAuth validation before execution (database query)
Circuit breaker prevents infinite retries
Deploy to Render and verify no scheduler startup
Monitor memory usage (should stay < 100MB)
Test manual "check my email" command (should still work)
Enable scheduler with env var and verify OAuth check

Production Schedule: 3x Daily (Cost-Optimized) ✅¶

Schedule enabled with: - 8:00 AM - Morning email check (catch overnight emails) - 1:00 PM - Midday email check (catch morning emails) - 6:00 PM - Evening email check (catch afternoon emails)

Cost: $0.18/day for 300 users = **$5.40/month total**

To enable scheduler, set environment variable in Render:

ENABLE_PROACTIVE_SCHEDULER=true

To disable scheduler:

ENABLE_PROACTIVE_SCHEDULER=false

Verify OAuth tokens in database:

SELECT user_phone, scope, expires_at
FROM oauth_tokens
WHERE expires_at > NOW();

Monitor logs for:
"Proactive scheduler started"
"Found N active users with valid OAuth tokens"
Circuit breaker status
Monitor metrics:
Memory usage (should stay < 150MB)
LLM token usage (should be ~400 tokens per run)
No repeated messages to same user

Lessons Learned¶

Never hardcode user IDs in production code
Always add kill switches for background services
Set memory/token budgets for LLM features
Add circuit breakers for external API calls
Test resource usage before deploying to production

app/main.py - Added kill switch
app/scheduler/background_service.py - OAuth validation, circuit breaker, fixed user query
app/superpowers/catalog/proactive/email_urgency.py - Reduced memory/LLM usage

Commit: "fix: Gmail spam & memory leak - add circuit breaker, OAuth validation, reduce costs"