Deployment Guide - Archety Backend

Complete guide for deploying the Archety backend to dev and production environments.

---

Table of Contents

1. Overview
2. Prerequisites
3. Environment Setup
4. Deployment Workflows
5. Rollback Procedures
6. Monitoring & Debugging
7. Common Issues
8. Emergency Procedures

---

Overview

Architecture

graph LR
    GitHub[GitHub Repository] --> Actions[GitHub Actions]
    Actions --> RailwayDev[Railway Dev]
    Actions --> RailwayProd[Railway Prod]

    subgraph "Environments"
    RailwayDev
    RailwayProd
    end

Environments

• :material-dev-to-app:{ .lg .middle } Development

  ---

  Auto-deploys from dev branch.

  View Dev Env

• Production

  ---

  Manual deploy from master (canonical domain is typically https://api.ikiro.ai).

  View Prod Env

---

Prerequisites

Required Access

• GitHub Access

  Write permissions required for the repository.

• Railway Account

  Project access and CLI installed (npm install -g @railway/cli).

• Secrets Configured

  See Railway Setup Guide.

Required Secrets

Configure in GitHub Settings

Go to Settings → Secrets and variables → Actions to configure these.

Secret Name	Description	Source
RAILWAY_TOKEN	Railway API token	railway token
RAILWAY_DEV_SERVICE_ID	Dev service ID	railway service (select dev)
RAILWAY_PROD_SERVICE_ID	Prod service ID	railway service (select prod)
OPENAI_API_KEY_DEV	Dev OpenAI key (tests)	OpenAI Platform
OPENAI_API_KEY_PROD	Prod OpenAI key (tests)	OpenAI Platform
MEM0_API_KEY_DEV	Dev mem0 key (tests; legacy memory)	mem0 dashboard
MEM0_API_KEY_PROD	Prod mem0 key (tests; legacy memory)	mem0 dashboard
PERPLEXITY_API_KEY_DEV	Dev Perplexity key (optional tests)	Perplexity dashboard

---

Environment Setup

Initial Setup

If you haven't set up Railway environments yet, follow the Railway Setup Guide first.

Quick Verification

# Verify Railway CLI is installed
railway --version

# Verify you're logged in
railway whoami

# Check current service
railway status

---

Deployment Workflows

1. Deploy to Development (Automatic)

Trigger: Push to dev branch

Steps:

# Create or checkout dev branch
git checkout dev

# Make your changes
git add .
git commit -m "feat: add new feature"

# Push to dev branch
git push origin dev

What Happens:

1. ✅ GitHub Actions runs tests
2. ✅ If tests pass, auto-deploys to Railway dev service
3. ✅ Health check verifies deployment
4. ✅ You get notified via GitHub Actions

Check Deployment:

# View GitHub Actions run
# Go to: https://github.com/YOUR_REPO/actions

# Check Railway logs
railway service  # Select dev
railway logs

2. Deploy to Production (Manual)

Method A: Manual Workflow Dispatch

1. Go to GitHub → Actions → "Deploy to Production"
2. Click "Run workflow"
3. Type DEPLOY TO PRODUCTION (exactly) in the confirmation field
4. Click "Run workflow"

Method B: Version Tag

# Make sure master is up to date
git checkout master
git merge dev
git push origin master

# Create a version tag
git tag v1.0.0
git push origin v1.0.0

What Happens:

1. ✅ GitHub Actions validates request
2. ✅ Runs full test suite
3. ✅ Checks for exposed secrets
4. ⏸️ Manual approval required (GitHub environment protection)
5. ✅ Deploys to Railway prod service
6. ✅ Health check verifies deployment
7. ✅ Creates deployment record on GitHub

Check Deployment:

# View GitHub Actions run
# Go to: https://github.com/YOUR_REPO/actions

# Check Railway logs
railway service  # Select prod
railway logs

# Run smoke tests
ENVIRONMENT_URL=https://archety-backend-prod.up.railway.app python tests/smoke/smoke_tests.py

3. Hotfix to Production (Emergency)

When: Critical bug in production that needs immediate fix

Steps:

# Create hotfix branch from master
git checkout master
git pull origin master
git checkout -b hotfix/critical-bug-fix

# Make the fix
git add .
git commit -m "hotfix: fix critical bug"

# Push and create PR
git push origin hotfix/critical-bug-fix

# Create PR targeting master
# After review and tests pass, merge PR

# Deploy immediately
git checkout master
git pull origin master
git tag v1.0.1-hotfix
git push origin v1.0.1-hotfix

---

Rollback Procedures

Rollback via Railway Dashboard

Fastest method for production issues:

1. Go to Railway dashboard
2. Select production service
3. Click on "Deployments" tab
4. Find the last known good deployment
5. Click "Redeploy"

Time: ~2-3 minutes

Rollback via CLI

# Switch to production service
railway service  # Select prod

# View recent deployments
railway logs

# Redeploy previous version
railway redeploy <DEPLOYMENT_ID>

Rollback via Git

# Revert the problematic commit
git revert <BAD_COMMIT_SHA>

# Push to master
git push origin master

# Deploy (manual trigger or tag)
git tag v1.0.2-revert
git push origin v1.0.2-revert

---

Monitoring & Debugging

Health Checks

Dev:

curl https://archety-backend-dev.up.railway.app/health

Prod:

curl https://archety-backend-prod.up.railway.app/health

Expected response:

{
  "status": "healthy",
  "timestamp": "2026-01-01T12:00:00Z"
}

View Logs

Railway Dashboard: 1. Go to Railway project 2. Select service (dev or prod) 3. Click "Logs" tab 4. Filter by level, time, etc.

Railway CLI:

# Dev logs
railway service  # Select dev
railway logs --follow

# Prod logs
railway service  # Select prod
railway logs --tail 100

Run Smoke Tests

# Test dev
ENVIRONMENT_URL=https://archety-backend-dev.up.railway.app \
  python tests/smoke/smoke_tests.py

# Test prod
ENVIRONMENT_URL=https://archety-backend-prod.up.railway.app \
  python tests/smoke/smoke_tests.py

Check Sentry for Errors

1. Go to Sentry Dashboard
2. Filter by environment (dev or production)
3. Review recent errors
4. Check error trends

Database Health

# Connect to dev database
railway service  # Select dev
railway connect postgres

# Connect to prod database
railway service  # Select prod
railway connect postgres

# Check connections
SELECT count(*) FROM pg_stat_activity;

# Check table sizes
SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;

---

Common Issues

Issue: Deployment Failed - Tests Not Passing

Symptoms: - GitHub Actions shows red X - Tests fail in CI

Solution:

# Run tests locally first
python -m pytest tests/ -v

# Fix any failing tests
git add .
git commit -m "fix: resolve test failures"
git push origin dev

Issue: Health Check Failing After Deployment

Symptoms: - Deployment succeeds but health check returns 500 - Application crashes on startup

Check:

# View logs immediately
railway logs --tail 100

# Common causes:
# 1. Missing environment variables
railway variables

# 2. Database migration failed
railway run alembic upgrade head

# 3. Database connection issues
railway connect postgres  # Check if database is accessible

Solution:

# Add missing variables
railway variables set MISSING_KEY=value

# Force redeploy
railway redeploy

Issue: Database Migration Failed

Symptoms: - Logs show "migration failed" - Database schema out of sync

Solution:

# Check current migration version
railway run alembic current

# Check migration history
railway run alembic history

# Try manual migration
railway run alembic upgrade head

# If stuck, downgrade and retry
railway run alembic downgrade -1
railway run alembic upgrade head

Issue: Environment Variables Not Updating

Symptoms: - Changed variables in Railway UI - Application still using old values

Solution:

# Redeploy service (triggers reload of env vars)
railway redeploy

# Or, restart service
railway run -- /bin/sh -c "kill 1"

Issue: CORS Errors in Frontend

Symptoms: - Frontend gets CORS errors - Browser console shows origin not allowed

Solution:

# Check BASE_URL is set correctly
railway variables | grep BASE_URL

# Update BASE_URL for environment
railway variables set BASE_URL=https://archety-backend-prod.up.railway.app

# Redeploy
railway redeploy

Issue: Out of Memory (OOM)

Symptoms: - Service crashes randomly - Logs show "Killed" or OOM errors

Solution:

1. Upgrade Railway plan (if on free tier)
2. Optimize memory usage:

   # Check current memory usage
   railway logs | grep memory
   
   # Reduce worker count in production
   # (edit start_with_migrations.sh)
   

---

Emergency Procedures

🚨 Service is Down

Priority: CRITICAL

Steps:

1. Check health endpoint:

   curl https://archety-backend-prod.up.railway.app/health
   

2. Check Railway status:

3. Go to Railway dashboard

4. Check if service is running

5. Check logs for crashes:

   railway logs --tail 50
   

6. Rollback immediately:

   # Use Railway dashboard to redeploy last known good version
   # OR
   railway redeploy <LAST_GOOD_DEPLOYMENT_ID>
   

7. Notify team:

8. Post in team chat
9. Update status page (if applicable)

🚨 Database Connection Lost

Priority: HIGH

Steps:

1. Check database status:

   railway connect postgres
   

2. Check database logs:

3. Go to Railway dashboard
4. Select PostgreSQL service

5. View logs

6. Restart database if needed:

7. Railway dashboard → PostgreSQL → Settings → Restart

8. Restart web service:

   railway redeploy
   

🚨 Exposed Secrets in Code

Priority: CRITICAL

Steps:

1. Immediately rotate all exposed secrets:

   # Generate new keys
   openssl rand -hex 32  # For SECRET_KEY
   python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"  # For FERNET_KEY
   

2. Update Railway environment variables:

   railway variables set SECRET_KEY=<new_key>
   railway variables set FERNET_KEY=<new_key>
   railway variables set EDGE_SECRET=<new_key>
   

3. Rotate external API keys:

4. Supermemory: Generate new key at supermemory.ai dashboard
5. mem0 (if using legacy fallback): Generate new key in dashboard
6. OpenAI: Rotate key in platform

7. OAuth: Update Google Console credentials

8. Remove secrets from git history:

   # Use git-filter-repo or BFG Repo-Cleaner
   # Contact DevOps team for assistance
   

9. Audit access logs for potential unauthorized usage

---

Deployment Checklist

Pre-Deployment

• All tests passing locally
• Code reviewed and approved
• Environment variables verified
• Database migrations tested
• Changelog updated

During Deployment

• Monitor GitHub Actions
• Watch Railway logs during deployment
• Health check passes
• Smoke tests pass

Post-Deployment

• Verify key user flows work
• Check Sentry for new errors
• Monitor performance metrics
• Update team on deployment status
• Tag release in GitHub (if production)

---

Additional Resources

• Railway Setup Guide
• Architecture Documentation
• API Documentation
• Environment Variables Guide

---

Last Updated: 2025-01-11 Maintained by: Engineering Team