Files
CHORUS/deploy/seqthink/test-e2e.sh
anthonyrawlins c99def17d7 Add deployment infrastructure and documentation
This commit adds complete deployment infrastructure for the Sequential Thinking
Age-Encrypted Wrapper, ready for Docker Swarm production deployment.

## Deliverables

### 1. Docker Swarm Service Definition
**File**: `deploy/seqthink/docker-compose.swarm.yml`

**Features**:
- 3 replicas with automatic spreading across worker nodes
- Resource limits: 1 CPU / 512MB RAM per replica
- Resource reservations: 0.5 CPU / 256MB RAM per replica
- Rolling updates with automatic rollback
- Health checks every 30 seconds
- Traefik integration with automatic HTTPS
- Load balancer with health checking
- Docker Secrets integration for age keys
- Comprehensive logging configuration

**Environment Variables**:
- `LOG_LEVEL`: Logging verbosity
- `MCP_LOCAL`: MCP server URL (loopback only)
- `PORT`: HTTP server port (8443)
- `MAX_BODY_MB`: Request size limit
- `AGE_IDENT_PATH`: Age private key path
- `AGE_RECIPS_PATH`: Age public key path
- `KACHING_JWKS_URL`: KACHING JWKS endpoint
- `REQUIRED_SCOPE`: Required JWT scope

**Secrets**:
- `seqthink_age_identity`: Age private key (mounted at /run/secrets/)
- `seqthink_age_recipients`: Age public key (mounted at /run/secrets/)

**Network**:
- `chorus-overlay`: External overlay network for service mesh

**Labels**:
- Traefik routing: `seqthink.chorus.services`
- HTTPS with Let's Encrypt
- Health check path: `/health`
- Load balancer port: 8443

### 2. Deployment Documentation
**File**: `deploy/seqthink/DEPLOYMENT.md` (500+ lines)

**Sections**:
1. **Prerequisites**: Cluster setup, network requirements
2. **Architecture**: Security layers diagram
3. **Step-by-step deployment**:
   - Generate age keys
   - Create Docker secrets
   - Build Docker image
   - Deploy to swarm
   - Verify deployment
   - Test with JWT tokens
4. **Configuration reference**: All environment variables documented
5. **Scaling**: Horizontal scaling commands
6. **Updates**: Rolling update procedures
7. **Rollback**: Automatic and manual rollback
8. **Monitoring**: Prometheus metrics, health checks, logs
9. **Troubleshooting**: Common issues and solutions
10. **Security considerations**: Key rotation, TLS, rate limiting
11. **Development mode**: Testing without security
12. **Production checklist**: Pre-deployment verification

### 3. End-to-End Test Script
**File**: `deploy/seqthink/test-e2e.sh` (executable)

**Tests**:
1. Health endpoint validation
2. Readiness endpoint validation
3. Metrics endpoint verification
4. Unauthorized request rejection (401)
5. Invalid authorization header rejection (401)
6. JWT token validation (if token provided)
7. Encrypted request/response (if age keys provided)
8. Content-Type validation (415 for wrong type)
9. Metrics collection verification
10. SSE endpoint availability

**Usage**:
```bash
# Basic tests (no auth)
./deploy/seqthink/test-e2e.sh

# With JWT token
export JWT_TOKEN="eyJhbGci..."
./deploy/seqthink/test-e2e.sh

# With JWT + encryption
export JWT_TOKEN="eyJhbGci..."
export AGE_RECIPIENT="$(cat seqthink_age.pub)"
export AGE_IDENTITY="seqthink_age.key"
./deploy/seqthink/test-e2e.sh
```

**Output**:
- Color-coded test results (✓ pass, ✗ fail, ⚠ warn)
- Test summary with counts
- Exit code 0 if all pass, 1 if any fail

### 4. Secrets Management Guide
**File**: `deploy/seqthink/SECRETS.md` (400+ lines)

**Topics**:
1. **Secret types**: Age keys, KACHING config
2. **Key generation**:
   - Method 1: Using age-keygen
   - Method 2: Using Go code
3. **Storing secrets**: Docker Swarm secret commands
4. **Using secrets**: Compose file configuration
5. **Key rotation**:
   - Why rotate
   - 5-step rotation process
   - Zero-downtime rotation
6. **Backup and recovery**:
   - Secure backup procedures
   - Age-encrypted backups
   - Recovery process
7. **Security best practices**:
   - Key generation ✓/✗ guidelines
   - Key storage ✓/✗ guidelines
   - Key distribution ✓/✗ guidelines
   - Key lifecycle ✓/✗ guidelines
8. **Troubleshooting**: Common secret issues
9. **Client-side key management**: Distributing public keys
10. **Compliance and auditing**: SOC 2, ISO 27001
11. **Emergency procedures**: Key compromise response

## Deployment Flow

### Initial Deployment
```bash
# 1. Generate keys
age-keygen -o seqthink_age.key
age-keygen -y seqthink_age.key > seqthink_age.pub

# 2. Create secrets
docker secret create seqthink_age_identity seqthink_age.key
docker secret create seqthink_age_recipients seqthink_age.pub

# 3. Build image
docker build -f deploy/seqthink/Dockerfile -t anthonyrawlins/seqthink-wrapper:latest .
docker push anthonyrawlins/seqthink-wrapper:latest

# 4. Deploy
docker stack deploy -c deploy/seqthink/docker-compose.swarm.yml seqthink

# 5. Verify
docker service ps seqthink_seqthink-wrapper
docker service logs seqthink_seqthink-wrapper

# 6. Test
./deploy/seqthink/test-e2e.sh
```

### Update Deployment
```bash
# Build new version
docker build -f deploy/seqthink/Dockerfile -t anthonyrawlins/seqthink-wrapper:0.2.0 .
docker push anthonyrawlins/seqthink-wrapper:0.2.0

# Rolling update
docker service update \
  --image anthonyrawlins/seqthink-wrapper:0.2.0 \
  seqthink_seqthink-wrapper
```

## Service Architecture

```
Internet
    ↓
Traefik (HTTPS + Load Balancing)
    ↓
seqthink.chorus.services
    ↓
Docker Swarm Overlay Network
    ↓
SeqThink Wrapper (3 replicas)
    ├─ JWT Validation (KACHING)
    ├─ Age Decryption
    ├─ MCP Server (loopback)
    ├─ Age Encryption
    └─ Response
```

## Security Layers

1. **Transport Security**: TLS 1.3 via Traefik
2. **Authentication**: JWT signature verification (RS256)
3. **Authorization**: Scope-based access control
4. **Encryption**: Age end-to-end encryption
5. **Network Isolation**: MCP server on loopback only
6. **Secrets Management**: Docker Swarm secrets (tmpfs)
7. **Resource Limits**: Container resource constraints

## Monitoring Integration

**Prometheus Metrics** (`/metrics`):
- `seqthink_requests_total`: Total requests
- `seqthink_errors_total`: Total errors
- `seqthink_decrypt_failures_total`: Decryption failures
- `seqthink_encrypt_failures_total`: Encryption failures
- `seqthink_policy_denials_total`: Authorization denials
- `seqthink_request_duration_seconds`: Request latency

**Health Checks**:
- `/health`: Liveness probe (wrapper running)
- `/ready`: Readiness probe (MCP server ready)

**Logging**:
- JSON format via docker logs
- 10MB max size, 3 file rotation
- Centralized log aggregation ready

## Production Readiness

 **High Availability**: 3 replicas with auto-restart
 **Zero-Downtime Updates**: Rolling updates with health checks
 **Automatic Rollback**: On update failure
 **Resource Management**: CPU/memory limits and reservations
 **Security**: Multi-layer defense (TLS, JWT, Age, Secrets)
 **Monitoring**: Metrics, health checks, structured logs
 **Documentation**: Complete deployment and operations guides
 **Testing**: Automated E2E test suite
 **Secrets Management**: Docker Swarm secrets with rotation procedures

## Next Steps

1. Test deployment on staging environment
2. Generate production age keys
3. Configure KACHING JWT integration
4. Deploy to production cluster
5. Monitor metrics and logs
6. Load testing and performance validation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 08:53:36 +11:00

217 lines
6.4 KiB
Bash
Executable File

#!/bin/bash
# End-to-end test script for Sequential Thinking Age Wrapper
set -e
echo "🧪 Sequential Thinking Wrapper E2E Tests"
echo "========================================"
echo ""
# Configuration
WRAPPER_URL="${WRAPPER_URL:-http://localhost:8443}"
JWT_TOKEN="${JWT_TOKEN:-}"
AGE_RECIPIENT="${AGE_RECIPIENT:-}"
AGE_IDENTITY="${AGE_IDENTITY:-}"
# Color codes
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Test counters
TESTS_RUN=0
TESTS_PASSED=0
TESTS_FAILED=0
# Helper functions
pass() {
echo -e "${GREEN}${NC} $1"
((TESTS_PASSED++))
}
fail() {
echo -e "${RED}${NC} $1"
((TESTS_FAILED++))
}
warn() {
echo -e "${YELLOW}${NC} $1"
}
test_start() {
((TESTS_RUN++))
echo ""
echo "Test $TESTS_RUN: $1"
echo "---"
}
# Test 1: Health Check
test_start "Health endpoint"
if curl -sf "$WRAPPER_URL/health" > /dev/null 2>&1; then
pass "Health check passed"
else
fail "Health check failed"
fi
# Test 2: Readiness Check
test_start "Readiness endpoint"
if curl -sf "$WRAPPER_URL/ready" > /dev/null 2>&1; then
pass "Readiness check passed"
else
fail "Readiness check failed"
fi
# Test 3: Metrics Endpoint
test_start "Metrics endpoint"
if curl -sf "$WRAPPER_URL/metrics" | grep -q "seqthink_requests_total"; then
pass "Metrics endpoint accessible"
else
fail "Metrics endpoint failed"
fi
# Test 4: Unauthorized Request (no token)
test_start "Unauthorized request rejection"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \
-H "Content-Type: application/json" \
-d '{"tool":"test"}')
if [ "$HTTP_CODE" = "401" ]; then
pass "Unauthorized request correctly rejected (401)"
else
warn "Expected 401, got $HTTP_CODE (may be policy disabled)"
fi
# Test 5: Invalid Authorization Header
test_start "Invalid authorization header"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \
-H "Authorization: InvalidFormat" \
-H "Content-Type: application/json" \
-d '{"tool":"test"}')
if [ "$HTTP_CODE" = "401" ]; then
pass "Invalid auth header correctly rejected (401)"
else
warn "Expected 401, got $HTTP_CODE (may be policy disabled)"
fi
# Test 6: JWT Token Validation (if token provided)
if [ -n "$JWT_TOKEN" ]; then
test_start "JWT token validation"
# Check if age keys are available
if [ -n "$AGE_RECIPIENT" ] && [ -n "$AGE_IDENTITY" ]; then
# Test with encryption
test_start "Encrypted request with valid JWT"
# Create test payload
TEST_PAYLOAD='{"tool":"mcp__sequential-thinking__sequentialthinking","payload":{"thought":"Test thought","thoughtNumber":1,"totalThoughts":1,"nextThoughtNeeded":false}}'
# Encrypt payload
ENCRYPTED_PAYLOAD=$(echo "$TEST_PAYLOAD" | age -r "$AGE_RECIPIENT" 2>/dev/null)
if [ $? -eq 0 ]; then
# Send encrypted request
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \
-H "Authorization: Bearer $JWT_TOKEN" \
-H "Content-Type: application/age" \
-d "$ENCRYPTED_PAYLOAD")
if [ "$HTTP_CODE" = "200" ]; then
pass "Encrypted request with JWT succeeded"
else
fail "Encrypted request failed with HTTP $HTTP_CODE"
fi
else
fail "Failed to encrypt test payload"
fi
else
# Test without encryption (plaintext mode)
test_start "Plaintext request with valid JWT"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \
-H "Authorization: Bearer $JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{"tool":"mcp__sequential-thinking__sequentialthinking","payload":{"thought":"Test","thoughtNumber":1,"totalThoughts":1,"nextThoughtNeeded":false}}')
if [ "$HTTP_CODE" = "200" ]; then
pass "Plaintext request with JWT succeeded"
else
warn "Request failed with HTTP $HTTP_CODE"
fi
fi
else
warn "JWT_TOKEN not set - skipping authenticated tests"
fi
# Test 7: Content-Type Validation (if encryption enabled)
if [ -n "$AGE_RECIPIENT" ]; then
test_start "Content-Type validation for encrypted mode"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \
-H "Authorization: Bearer ${JWT_TOKEN:-dummy}" \
-H "Content-Type: application/json" \
-d '{"tool":"test"}')
if [ "$HTTP_CODE" = "415" ]; then
pass "Incorrect Content-Type correctly rejected (415)"
else
warn "Expected 415, got $HTTP_CODE"
fi
fi
# Test 8: Metrics Collection
test_start "Metrics collection"
METRICS=$(curl -s "$WRAPPER_URL/metrics")
if echo "$METRICS" | grep -q "seqthink_requests_total"; then
REQUEST_COUNT=$(echo "$METRICS" | grep "^seqthink_requests_total" | awk '{print $2}')
pass "Request metrics collected (total: $REQUEST_COUNT)"
else
fail "Request metrics not found"
fi
if echo "$METRICS" | grep -q "seqthink_errors_total"; then
ERROR_COUNT=$(echo "$METRICS" | grep "^seqthink_errors_total" | awk '{print $2}')
pass "Error metrics collected (total: $ERROR_COUNT)"
else
fail "Error metrics not found"
fi
if echo "$METRICS" | grep -q "seqthink_policy_denials_total"; then
DENIAL_COUNT=$(echo "$METRICS" | grep "^seqthink_policy_denials_total" | awk '{print $2}')
pass "Policy denial metrics collected (total: $DENIAL_COUNT)"
else
warn "Policy denial metrics not found (may be policy disabled)"
fi
# Test 9: SSE Endpoint (basic check)
test_start "SSE endpoint availability"
# Just check if endpoint exists, don't try to consume stream
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 2 "$WRAPPER_URL/mcp/sse" 2>/dev/null || echo "timeout")
if [ "$HTTP_CODE" = "401" ] || [ "$HTTP_CODE" = "200" ]; then
pass "SSE endpoint exists (HTTP $HTTP_CODE)"
else
warn "SSE endpoint check inconclusive (HTTP $HTTP_CODE)"
fi
# Summary
echo ""
echo "========================================"
echo "Test Summary"
echo "========================================"
echo "Tests Run: $TESTS_RUN"
echo -e "${GREEN}Tests Passed: $TESTS_PASSED${NC}"
if [ $TESTS_FAILED -gt 0 ]; then
echo -e "${RED}Tests Failed: $TESTS_FAILED${NC}"
fi
echo ""
if [ $TESTS_FAILED -eq 0 ]; then
echo -e "${GREEN}✓ All tests passed!${NC}"
exit 0
else
echo -e "${RED}✗ Some tests failed${NC}"
exit 1
fi