Keywords AI Full Request/Response Logging

Current Status

What's Installed

✅ keywordsai-tracing v0.0.51 - Installed and initialized ✅ OpenTelemetry instrumentation - Automatically instruments OpenAI calls ✅ Integration active - Initialized on app startup (app/main.py:117-119)

Current Configuration Issue

❌ CRITICAL: Prompts and completions are NOT being captured by default!

Keywords AI uses OpenTelemetry's opentelemetry-instrumentation-openai which, by default, only captures: - Model name - Token counts (prompt_tokens, completion_tokens, total_tokens) - Duration/latency - Cost - Request metadata

It does NOT capture: - ❌ System prompts - ❌ User messages/prompts - ❌ LLM completions/responses - ❌ Conversation history - ❌ Retrieved memories

Why This Matters

According to OpenTelemetry documentation:

"Message content such as the contents of the prompt, completion, function arguments and return values are not captured by default."

To capture full content, you must set:

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

Solution

1. Enable Full Content Capture

Add the following environment variable to enable full prompt/completion logging:

# Enable full LLM request/response content capture
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

⚠️ Privacy Warning: This will capture ALL LLM input/output including: - User messages (potentially containing PII) - System prompts (containing memories, relationship state, etc.) - Complete responses

Ensure compliance with your privacy policy and data retention requirements.

2. What Will Be Captured

With this setting enabled, Keywords AI will log:

Full OpenAI Request:

• ✅ Model name (e.g., "gpt-5", "gpt-5-mini")
• ✅ Complete system prompt (persona, style, context)
• ✅ Complete user message (including conversation history)
• ✅ All retrieved memories included in context
• ✅ Temperature, max_tokens, and other parameters

Full OpenAI Response:

• ✅ Complete LLM response text
• ✅ Finish reason
• ✅ Token counts (prompt, completion, reasoning, total)
• ✅ Cost calculation
• ✅ Latency/duration

Multi-Stage Traces:

• ✅ Classifier call (intent detection)
• ✅ Reflex generation
• ✅ Main response generation
• ✅ Memory retrieval operations
• ✅ Workflow execution

3. Current Integration Points

The system currently instruments these LLM calls:

1. app/utils/llm_client.py - All OpenAI client calls are automatically instrumented
2. _openai_generate() - Main response generation
3. call_gpt5() - Reasoning model calls
4. call_gpt5_mini() - Fast classification calls

5. classify_intent() - Intent detection

6. app/orchestrator/two_stage_handler.py - 2-stage pipeline traces

7. Stage 1: Classification (GPT-5-mini)

8. Stage 2: Generation (GPT-5)

9. Custom decorators - app/utils/tracing.py

10. @workflow() - High-level agent workflows
11. @task() - Individual LLM operations

4. Accessing Logged Data

Keywords AI provides several ways to access logged data:

Via Keywords AI Dashboard:

1. Log into https://platform.keywordsai.co
2. Navigate to "Logs" section
3. View full request/response traces
4. Filter by user_id, model, date range
5. Export to CSV/JSON

Via API (for programmatic access):

from keywordsai_sdk import KeywordsAI

client = KeywordsAI(api_key="your_key")

# Get logs for a specific time range
logs = client.logs.list(
    start_date="2025-01-01",
    end_date="2025-01-31",
    user_id="user_123"  # Optional filter
)

# Export to file
for log in logs:
    print(f"Request: {log.prompt_messages}")
    print(f"Response: {log.completion_message}")
    print(f"Cost: ${log.cost}")
    print(f"Tokens: {log.total_tokens}")

5. Search and Analytics

Keywords AI provides: - Full-text search across all prompts and responses - Request filtering by model, user, date, cost, latency - Cost analytics - breakdown by user, model, feature - Latency analysis - P50, P90, P95, P99 metrics - Error tracking - failed requests, refusals, timeouts

6. Data Retention

Keywords AI retains logs according to your plan: - Free tier: 7 days - Paid plans: 30-90 days (configurable)

For longer retention, set up automated exports.

7. Privacy and Compliance

When enabling full content capture:

1. Update Privacy Policy - Disclose that LLM conversations are logged
2. Implement Data Retention - Set up automated deletion after N days
3. PII Handling - Consider implementing PII redaction before logging
4. User Consent - Ensure users are aware and consent to logging

8. Testing the Integration

After enabling OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true, verify:

# Test script to verify full capture
import os
os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "true"

from app.utils.llm_client import get_llm_client
from app.utils.tracing import initialize_tracing

# Initialize tracing
initialize_tracing()

# Make a test call
llm = get_llm_client()
response = llm.generate_response(
    system_prompt="You are a helpful assistant named Sage.",
    user_message="Hello! How are you today?",
    max_tokens=100
)

print(f"Response: {response}")
print("\n✅ Check Keywords AI dashboard for full request/response logs")

Then check your Keywords AI dashboard to confirm you see: - Full system prompt: "You are a helpful assistant named Sage." - Full user message: "Hello! How are you today?" - Full response text

Comparison with Other Solutions

Feature	Keywords AI	LangSmith	Helicone	Custom DB
Auto-instrumentation	✅ Yes	✅ Yes	⚠️ Proxy	❌ Manual
Full prompt logging	✅ With env var	✅ Yes	✅ Yes	✅ Yes
Multi-stage traces	✅ Yes	✅ Yes	❌ No	⚠️ Manual
Cost tracking	✅ Yes	✅ Yes	✅ Yes	⚠️ Manual
Export API	✅ Yes	✅ Yes	✅ Yes	✅ Yes
Data sovereignty	❌ External	❌ External	❌ External	✅ Self-hosted

Recommendation

✅ Enable Keywords AI full logging immediately by setting the environment variable. This provides: 1. Zero code changes required 2. Full request/response capture 3. Built-in analytics and search 4. Easy export capabilities 5. Already integrated and working

Next Steps

1. ✅ Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
2. ✅ Add to .env.example for documentation
3. ✅ Deploy to staging/production
4. ✅ Verify logs appear in Keywords AI dashboard
5. ✅ Set up regular export routine if needed for compliance
6. ✅ Update privacy policy to disclose logging

---

Created: 2025-01-14 Status: Implementation Required Owner: Engineering Team