Files

anthonyrawlins c99def17d7 Add deployment infrastructure and documentation

This commit adds complete deployment infrastructure for the Sequential Thinking
Age-Encrypted Wrapper, ready for Docker Swarm production deployment.

## Deliverables

### 1. Docker Swarm Service Definition
**File**: `deploy/seqthink/docker-compose.swarm.yml`

**Features**:
- 3 replicas with automatic spreading across worker nodes
- Resource limits: 1 CPU / 512MB RAM per replica
- Resource reservations: 0.5 CPU / 256MB RAM per replica
- Rolling updates with automatic rollback
- Health checks every 30 seconds
- Traefik integration with automatic HTTPS
- Load balancer with health checking
- Docker Secrets integration for age keys
- Comprehensive logging configuration

**Environment Variables**:
- `LOG_LEVEL`: Logging verbosity
- `MCP_LOCAL`: MCP server URL (loopback only)
- `PORT`: HTTP server port (8443)
- `MAX_BODY_MB`: Request size limit
- `AGE_IDENT_PATH`: Age private key path
- `AGE_RECIPS_PATH`: Age public key path
- `KACHING_JWKS_URL`: KACHING JWKS endpoint
- `REQUIRED_SCOPE`: Required JWT scope

**Secrets**:
- `seqthink_age_identity`: Age private key (mounted at /run/secrets/)
- `seqthink_age_recipients`: Age public key (mounted at /run/secrets/)

**Network**:
- `chorus-overlay`: External overlay network for service mesh

**Labels**:
- Traefik routing: `seqthink.chorus.services`
- HTTPS with Let's Encrypt
- Health check path: `/health`
- Load balancer port: 8443

### 2. Deployment Documentation
**File**: `deploy/seqthink/DEPLOYMENT.md` (500+ lines)

**Sections**:
1. **Prerequisites**: Cluster setup, network requirements
2. **Architecture**: Security layers diagram
3. **Step-by-step deployment**:
   - Generate age keys
   - Create Docker secrets
   - Build Docker image
   - Deploy to swarm
   - Verify deployment
   - Test with JWT tokens
4. **Configuration reference**: All environment variables documented
5. **Scaling**: Horizontal scaling commands
6. **Updates**: Rolling update procedures
7. **Rollback**: Automatic and manual rollback
8. **Monitoring**: Prometheus metrics, health checks, logs
9. **Troubleshooting**: Common issues and solutions
10. **Security considerations**: Key rotation, TLS, rate limiting
11. **Development mode**: Testing without security
12. **Production checklist**: Pre-deployment verification

### 3. End-to-End Test Script
**File**: `deploy/seqthink/test-e2e.sh` (executable)

**Tests**:
1. Health endpoint validation
2. Readiness endpoint validation
3. Metrics endpoint verification
4. Unauthorized request rejection (401)
5. Invalid authorization header rejection (401)
6. JWT token validation (if token provided)
7. Encrypted request/response (if age keys provided)
8. Content-Type validation (415 for wrong type)
9. Metrics collection verification
10. SSE endpoint availability

**Usage**:
```bash
# Basic tests (no auth)
./deploy/seqthink/test-e2e.sh

# With JWT token
export JWT_TOKEN="eyJhbGci..."
./deploy/seqthink/test-e2e.sh

# With JWT + encryption
export JWT_TOKEN="eyJhbGci..."
export AGE_RECIPIENT="$(cat seqthink_age.pub)"
export AGE_IDENTITY="seqthink_age.key"
./deploy/seqthink/test-e2e.sh
```

**Output**:
- Color-coded test results (✓ pass, ✗ fail, ⚠ warn)
- Test summary with counts
- Exit code 0 if all pass, 1 if any fail

### 4. Secrets Management Guide
**File**: `deploy/seqthink/SECRETS.md` (400+ lines)

**Topics**:
1. **Secret types**: Age keys, KACHING config
2. **Key generation**:
   - Method 1: Using age-keygen
   - Method 2: Using Go code
3. **Storing secrets**: Docker Swarm secret commands
4. **Using secrets**: Compose file configuration
5. **Key rotation**:
   - Why rotate
   - 5-step rotation process
   - Zero-downtime rotation
6. **Backup and recovery**:
   - Secure backup procedures
   - Age-encrypted backups
   - Recovery process
7. **Security best practices**:
   - Key generation ✓/✗ guidelines
   - Key storage ✓/✗ guidelines
   - Key distribution ✓/✗ guidelines
   - Key lifecycle ✓/✗ guidelines
8. **Troubleshooting**: Common secret issues
9. **Client-side key management**: Distributing public keys
10. **Compliance and auditing**: SOC 2, ISO 27001
11. **Emergency procedures**: Key compromise response

## Deployment Flow

### Initial Deployment
```bash
# 1. Generate keys
age-keygen -o seqthink_age.key
age-keygen -y seqthink_age.key > seqthink_age.pub

# 2. Create secrets
docker secret create seqthink_age_identity seqthink_age.key
docker secret create seqthink_age_recipients seqthink_age.pub

# 3. Build image
docker build -f deploy/seqthink/Dockerfile -t anthonyrawlins/seqthink-wrapper:latest .
docker push anthonyrawlins/seqthink-wrapper:latest

# 4. Deploy
docker stack deploy -c deploy/seqthink/docker-compose.swarm.yml seqthink

# 5. Verify
docker service ps seqthink_seqthink-wrapper
docker service logs seqthink_seqthink-wrapper

# 6. Test
./deploy/seqthink/test-e2e.sh
```

### Update Deployment
```bash
# Build new version
docker build -f deploy/seqthink/Dockerfile -t anthonyrawlins/seqthink-wrapper:0.2.0 .
docker push anthonyrawlins/seqthink-wrapper:0.2.0

# Rolling update
docker service update \
  --image anthonyrawlins/seqthink-wrapper:0.2.0 \
  seqthink_seqthink-wrapper
```

## Service Architecture

```
Internet
    ↓
Traefik (HTTPS + Load Balancing)
    ↓
seqthink.chorus.services
    ↓
Docker Swarm Overlay Network
    ↓
SeqThink Wrapper (3 replicas)
    ├─ JWT Validation (KACHING)
    ├─ Age Decryption
    ├─ MCP Server (loopback)
    ├─ Age Encryption
    └─ Response
```

## Security Layers

1. **Transport Security**: TLS 1.3 via Traefik
2. **Authentication**: JWT signature verification (RS256)
3. **Authorization**: Scope-based access control
4. **Encryption**: Age end-to-end encryption
5. **Network Isolation**: MCP server on loopback only
6. **Secrets Management**: Docker Swarm secrets (tmpfs)
7. **Resource Limits**: Container resource constraints

## Monitoring Integration

**Prometheus Metrics** (`/metrics`):
- `seqthink_requests_total`: Total requests
- `seqthink_errors_total`: Total errors
- `seqthink_decrypt_failures_total`: Decryption failures
- `seqthink_encrypt_failures_total`: Encryption failures
- `seqthink_policy_denials_total`: Authorization denials
- `seqthink_request_duration_seconds`: Request latency

**Health Checks**:
- `/health`: Liveness probe (wrapper running)
- `/ready`: Readiness probe (MCP server ready)

**Logging**:
- JSON format via docker logs
- 10MB max size, 3 file rotation
- Centralized log aggregation ready

## Production Readiness

✅ **High Availability**: 3 replicas with auto-restart
✅ **Zero-Downtime Updates**: Rolling updates with health checks
✅ **Automatic Rollback**: On update failure
✅ **Resource Management**: CPU/memory limits and reservations
✅ **Security**: Multi-layer defense (TLS, JWT, Age, Secrets)
✅ **Monitoring**: Metrics, health checks, structured logs
✅ **Documentation**: Complete deployment and operations guides
✅ **Testing**: Automated E2E test suite
✅ **Secrets Management**: Docker Swarm secrets with rotation procedures

## Next Steps

1. Test deployment on staging environment
2. Generate production age keys
3. Configure KACHING JWT integration
4. Deploy to production cluster
5. Monitor metrics and logs
6. Load testing and performance validation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-10-13 08:53:36 +11:00

8.6 KiB

Raw Blame History

Sequential Thinking Age Wrapper - Deployment Guide

Overview

This guide covers deploying the Sequential Thinking Age-Encrypted Wrapper to Docker Swarm with full security enabled.

Prerequisites

Docker Swarm cluster initialized
chorus-overlay network created
Traefik reverse proxy configured
KACHING authentication service available

Architecture

Client → Traefik (HTTPS) → SeqThink Wrapper (JWT + Age Encryption) → MCP Server (loopback)

Security Layers:

TLS: Traefik terminates HTTPS
JWT: KACHING token validation
Age Encryption: End-to-end encrypted payloads

Step 1: Generate Age Keys

Generate a key pair for encryption:

# Generate age identity (private key)
age-keygen -o seqthink_age.key

# Extract recipient (public key)
age-keygen -y seqthink_age.key > seqthink_age.pub

Output:

seqthink_age.key:
# created: 2025-10-13T08:00:00+11:00
# public key: age1abcd...
AGE-SECRET-KEY-1ABCD...

seqthink_age.pub:
age1abcd...

Step 2: Create Docker Secrets

Store the age keys as Docker secrets:

# Create identity secret
docker secret create seqthink_age_identity seqthink_age.key

# Create recipient secret
docker secret create seqthink_age_recipients seqthink_age.pub

# Verify secrets
docker secret ls | grep seqthink

Expected Output:

seqthink_age_identity      <timestamp>
seqthink_age_recipients    <timestamp>

Step 3: Build Docker Image

Build the wrapper image:

cd /home/tony/chorus/project-queues/active/CHORUS

# Build image
docker build -f deploy/seqthink/Dockerfile -t anthonyrawlins/seqthink-wrapper:latest .

# Tag with version
docker tag anthonyrawlins/seqthink-wrapper:latest anthonyrawlins/seqthink-wrapper:0.1.0

# Push to registry
docker push anthonyrawlins/seqthink-wrapper:latest
docker push anthonyrawlins/seqthink-wrapper:0.1.0

Step 4: Deploy to Swarm

Deploy the service:

cd deploy/seqthink

# Deploy stack
docker stack deploy -c docker-compose.swarm.yml seqthink

# Check service status
docker service ls | grep seqthink

# Check logs
docker service logs -f seqthink_seqthink-wrapper

Expected Log Output:

🚀 Starting Sequential Thinking Age Wrapper
⏳ Waiting for MCP server...
✅ MCP server ready
Policy enforcement enabled
  jwks_url: https://auth.kaching.services/jwks
  required_scope: sequentialthinking.run
Fetching JWKS
JWKS cached successfully
  key_count: 2
Encryption enabled - using encrypted endpoint
🔐 Wrapper listening
  addr: :8443
  encryption_enabled: true
  policy_enabled: true

Step 5: Verify Deployment

Check service health:

# Check replicas
docker service ps seqthink_seqthink-wrapper

# Test health endpoint
curl -f http://localhost:8443/health
# Expected: OK

# Test readiness
curl -f http://localhost:8443/ready
# Expected: READY

# Check metrics
curl http://localhost:8443/metrics | grep seqthink

Step 6: Test with JWT Token

Get a KACHING JWT token and test the API:

# Set your JWT token
export JWT_TOKEN="eyJhbGciOiJSUzI1NiIsImtpZCI6ImRlZmF1bHQiLCJ0eXAiOiJKV1QifQ..."

# Test unauthorized (should fail)
curl -X POST https://seqthink.chorus.services/mcp/tool \
  -H "Content-Type: application/age" \
  -d "test"
# Expected: 401 Unauthorized

# Test authorized (should succeed)
curl -X POST https://seqthink.chorus.services/mcp/tool \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -H "Content-Type: application/age" \
  -d "$(echo '{"tool":"test","payload":{}}' | age -r $(cat seqthink_age.pub))" \
  --output encrypted_response.age

# Decrypt response
age -d -i seqthink_age.key encrypted_response.age

Configuration Reference

Environment Variables

Variable	Required	Default	Description
`PORT`	No	`8443`	HTTP server port
`MCP_LOCAL`	No	`http://127.0.0.1:8000`	MCP server URL (loopback)
`LOG_LEVEL`	No	`info`	Logging level (debug, info, warn, error)
`MAX_BODY_MB`	No	`4`	Maximum request body size in MB
`AGE_IDENT_PATH`	Yes	-	Path to age identity (private key)
`AGE_RECIPS_PATH`	Yes	-	Path to age recipients (public key)
`KACHING_JWKS_URL`	Yes	-	KACHING JWKS endpoint
`REQUIRED_SCOPE`	Yes	`sequentialthinking.run`	Required JWT scope

Docker Secrets

Secret Name	Purpose	Content
`seqthink_age_identity`	Age private key	`AGE-SECRET-KEY-1...`
`seqthink_age_recipients`	Age public key	`age1...`

Network Ports

Port	Protocol	Purpose
`8443`	HTTP	Wrapper API
`8000`	HTTP	MCP server (internal loopback only)

Scaling

Scale the service:

# Scale up
docker service scale seqthink_seqthink-wrapper=5

# Scale down
docker service scale seqthink_seqthink-wrapper=2

Updates

Rolling update:

# Build new version
docker build -f deploy/seqthink/Dockerfile -t anthonyrawlins/seqthink-wrapper:0.2.0 .
docker push anthonyrawlins/seqthink-wrapper:0.2.0

# Update service
docker service update \
  --image anthonyrawlins/seqthink-wrapper:0.2.0 \
  seqthink_seqthink-wrapper

# Monitor rollout
docker service ps seqthink_seqthink-wrapper

Rollback

If update fails:

# Automatic rollback (configured in stack)
# Or manual rollback:
docker service rollback seqthink_seqthink-wrapper

Monitoring

Prometheus Metrics

Available at http://localhost:8443/metrics:

seqthink_requests_total
seqthink_errors_total
seqthink_decrypt_failures_total
seqthink_encrypt_failures_total
seqthink_policy_denials_total
seqthink_request_duration_seconds

Health Checks

Liveness: GET /health - Returns 200 if wrapper is running
Readiness: GET /ready - Returns 200 if MCP server is ready

Logs

View logs:

# All replicas
docker service logs seqthink_seqthink-wrapper

# Follow logs
docker service logs -f seqthink_seqthink-wrapper

# Specific replica
docker service logs seqthink_seqthink-wrapper.<replica-id>

Troubleshooting

Issue: Policy Enforcement Disabled

Symptoms:

Policy enforcement disabled - no JWKS URL or required scope configured

Solution:

Verify KACHING_JWKS_URL and REQUIRED_SCOPE are set
Check environment variables: docker service inspect seqthink_seqthink-wrapper

Issue: JWKS Fetch Failed

Symptoms:

Failed to pre-fetch JWKS, will retry on first request

Solution:

Check KACHING service is accessible
Verify JWKS URL is correct
Check network connectivity

Issue: Decryption Failed

Symptoms:

Failed to decrypt request
seqthink_decrypt_failures_total increasing

Solution:

Verify age keys match between client and server
Check client is using correct public key
Ensure secrets are correctly mounted

Issue: MCP Server Not Ready

Symptoms:

❌ MCP server not ready
timeout waiting for MCP server

Solution:

Check MCP server is starting correctly
Review entrypoint.sh logs
Verify Python dependencies installed

Security Considerations

Key Rotation: Periodically rotate age keys:

# Generate new keys
age-keygen -o seqthink_age_new.key
age-keygen -y seqthink_age_new.key > seqthink_age_new.pub

# Update secrets (requires service restart)
docker secret rm seqthink_age_identity
docker secret create seqthink_age_identity seqthink_age_new.key

JWT Token Expiration: Tokens should have short expiration times (1 hour recommended)
Network Isolation: MCP server only accessible on loopback (127.0.0.1)
TLS: Always use HTTPS in production (via Traefik)
Rate Limiting: Consider adding rate limiting at Traefik level

Development Mode

For testing without security:

environment:
  # Disable encryption
  AGE_IDENT_PATH: ""
  AGE_RECIPS_PATH: ""

  # Disable policy
  KACHING_JWKS_URL: ""
  REQUIRED_SCOPE: ""

WARNING: Only use in development environments!

Production Checklist

Age keys generated and stored as Docker secrets
KACHING JWKS URL configured and accessible
Docker image built and pushed to registry
Service deployed to swarm
Health checks passing
Metrics endpoint accessible
JWT tokens validated successfully
End-to-end encryption verified
Logs show no errors
Monitoring alerts configured
Backup of age keys stored securely
Documentation updated with deployment details

Support

For issues or questions:

Check logs: docker service logs seqthink_seqthink-wrapper
Review metrics: curl http://localhost:8443/metrics
Consult implementation docs in /home/tony/chorus/project-queues/active/CHORUS/docs/

8.6 KiB Raw Blame History

Sequential Thinking Age Wrapper - Deployment Guide

Overview

Prerequisites

Architecture

Step 1: Generate Age Keys

Step 2: Create Docker Secrets

Step 3: Build Docker Image

Step 4: Deploy to Swarm

Step 5: Verify Deployment

Step 6: Test with JWT Token

Configuration Reference

Environment Variables

Docker Secrets

Network Ports

Scaling

Updates

Rollback

Monitoring

Prometheus Metrics

Health Checks

Logs

Troubleshooting

Issue: Policy Enforcement Disabled

Issue: JWKS Fetch Failed

Issue: Decryption Failed

Issue: MCP Server Not Ready

Security Considerations

Development Mode

Production Checklist

Support

8.6 KiB

Raw Blame History