Keywords AI Full Monitoring Implementation

Quick Setup

Step 1: Get Your Keywords AI API Key

1. Go to: https://platform.keywordsai.co/settings/api-keys
2. Create a new API key (or copy existing one)
3. It will look like: kw_xxxxxxxxxxxxxxxxxxxxxx

Step 2: Add to Your Environment

For Local Development — add to your .env file:

KEYWORDSAI_API_KEY=kw_xxxxxxxxxxxxxxxxxxxxxx
KEYWORDSAI_ENABLED=true
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

For Railway (Production):

1. Go to Railway dashboard → Variables
2. Add KEYWORDSAI_API_KEY, KEYWORDSAI_ENABLED=true, and OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

Step 3: Restart and Verify

# Run diagnostic
python scripts/diagnostic_keywords_ai.py

# Run test call
python scripts/test_keywords_ai_logging.py

Then check the dashboard: https://platform.keywordsai.co/logs

Success log on startup:

INFO: Keywords AI tracing initialized successfully with OpenAI instrumentation

Failure log (missing API key):

WARNING: Keywords AI API key not found - tracing will be disabled

Troubleshooting

• Set key but no logs? Run python scripts/diagnostic_keywords_ai.py and restart
• Dashboard empty? Make an actual LLM call first; logs take 5-10 seconds to appear
• Wrong workspace? Verify you're viewing the right project in Keywords AI

---

VERIFIED: Keywords AI Supports Full Request/Response Logging

After thorough investigation, Keywords AI CAN and WILL capture full chat requests and responses, including: - ✅ Complete system prompts - ✅ Full user messages - ✅ Complete LLM responses - ✅ All metadata (tokens, cost, latency) - ✅ Multi-stage traces (classifier → generator)

Critical Finding

❌ It is NOT enabled by default!

By default, Keywords AI only captures metadata (tokens, cost, latency) but NOT the actual prompt/response content.

Solution Implemented

1. Configuration Change Required

Add this environment variable to enable full content capture:

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

This has been added to: - ✅ .env.example with documentation (lines 164-167) - ✅ You need to add it to your actual .env file or Railway environment variables

2. How It Works

Current Setup: - ✅ keywordsai-tracing v0.0.51 is installed - ✅ Integration is active (initialized in app/main.py:117-119) - ✅ Uses OpenTelemetry to automatically instrument all OpenAI API calls

With Environment Variable Set: - ✅ Captures full system prompts (persona, style, memories) - ✅ Captures complete user messages and conversation history - ✅ Captures full LLM responses - ✅ Traces multi-stage pipeline (classifier → reflex → generator) - ✅ Logs all LLM calls across the codebase

Without Environment Variable (current state): - ⚠️ Only captures metadata: model name, tokens, cost, latency - ❌ Does NOT capture prompts or responses

What Gets Logged

Full OpenAI Request:

• Model: gpt-5, gpt-5-mini, gpt-5-nano
• System Prompt: Complete persona description, style guide, memories
• User Prompt: Full message including conversation history
• Parameters: temperature, max_tokens, reasoning_effort

Full OpenAI Response:

• Complete Response Text: Everything the LLM generated
• Finish Reason: stop, length, content_filter, etc.
• Token Breakdown: prompt_tokens, completion_tokens, reasoning_tokens, total_tokens
• Cost: Calculated in USD
• Latency: Time to first token, total time

Multi-Stage Traces:

• Stage 1: Intent Classification (GPT-5-mini)
• Stage 2: Response Generation (GPT-5)
• Memory Retrieval: Search queries and results
• Workflow Execution: Tool calls and results

Accessing the Data

Via Dashboard:

1. Log into: https://platform.keywordsai.co/logs
2. View full traces with prompts and responses
3. Filter by user, model, date, cost, latency
4. Export to CSV or JSON

Via API:

from keywordsai_sdk import KeywordsAI

client = KeywordsAI(api_key=os.getenv("KEYWORDSAI_API_KEY"))

logs = client.logs.list(
    start_date="2025-01-01",
    end_date="2025-01-31",
    user_id="user_123"  # Optional
)

for log in logs:
    print(f"Prompt: {log.prompt_messages}")
    print(f"Response: {log.completion_message}")
    print(f"Cost: ${log.cost}")

Testing

Run the test script to verify logging is working:

# Set the environment variable
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

# Run test
python scripts/test_keywords_ai_logging.py

This will: 1. Check environment configuration 2. Initialize tracing 3. Make a test LLM call 4. Provide instructions to verify in Keywords AI dashboard

Next Steps

1. Deploy to Production (REQUIRED)

Add to Railway environment variables:

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

2. Verify It Works

After deploying: 1. Make a test request through your app 2. Check Keywords AI dashboard 3. Confirm you see full prompts and responses

3. Set Up Regular Exports (Recommended)

For compliance and backup:

# Add to crontab for daily exports
0 2 * * * cd /path/to/archety && python scripts/export_daily_logs.py

4. Update Privacy Policy (REQUIRED)

Disclose that: - LLM conversations are logged for quality and debugging - Data is stored with Keywords AI (third-party service) - Users can request data export or deletion - Define retention period (e.g., 30 days)

5. Implement Data Retention

Set up automated deletion of old logs: - Via Keywords AI dashboard settings - Or via automated script to delete logs older than N days

Privacy Considerations

⚠️ WARNING: This captures ALL user data in LLM conversations

Including: - User messages (potentially containing PII) - System prompts (containing user memories, relationship state) - Complete conversation history

Required Actions: 1. ✅ Update Terms of Service / Privacy Policy 2. ✅ Obtain user consent for logging 3. ✅ Implement data retention policy 4. ✅ Support GDPR data export requests 5. ✅ Implement secure access controls

Documentation

Detailed guides created: - 📄 docs/archive/historical/KEYWORDS_AI_FULL_LOGGING.md - Complete setup guide (archived) - 📄 docs/archive/historical/KEYWORDS_AI_EXPORT_GUIDE.md - Export methods and automation (archived) - 📄 scripts/test_keywords_ai_logging.py - Test script

Comparison with Alternatives

Feature	Keywords AI	LangSmith	Helicone	Custom DB
Setup Effort	✅ 1 env var	⚠️ Moderate	⚠️ Proxy setup	❌ High
Auto-instrumentation	✅ Yes	✅ Yes	⚠️ Proxy	❌ Manual
Full logging	✅ Yes	✅ Yes	✅ Yes	✅ Yes
Multi-stage traces	✅ Yes	✅ Yes	❌ Limited	⚠️ Manual
Export API	✅ Yes	✅ Yes	✅ Yes	✅ Yes
Cost	💰 Paid	💰 Paid	💰 Paid	Free (hosting)

Recommendation

✅ Use Keywords AI - It's already integrated and just needs one environment variable!

Advantages: 1. Already installed and working 2. Zero code changes required 3. Just set one environment variable 4. Automatic instrumentation of all OpenAI calls 5. Full multi-stage trace support 6. Built-in analytics dashboard 7. Export API for compliance

Next Best Alternative: LangSmith (if you need more advanced features)

Summary

Status	Item
✅ VERIFIED	Keywords AI supports full request/response logging
✅ INSTALLED	keywordsai-tracing v0.0.51 is installed
✅ INTEGRATED	Initialization code is active in app/main.py
⚠️ PENDING	Environment variable needs to be set
✅ DOCUMENTED	Full guides and test scripts created
📋 TODO	Deploy env var to production
📋 TODO	Verify in dashboard after deployment
📋 TODO	Update privacy policy
📋 TODO	Set up automated exports

---

Created: 2025-01-14 Status: Ready for Deployment Action Required: Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true in production