This commit adds complete deployment infrastructure for the Sequential Thinking
Age-Encrypted Wrapper, ready for Docker Swarm production deployment.
## Deliverables
### 1. Docker Swarm Service Definition
**File**: `deploy/seqthink/docker-compose.swarm.yml`
**Features**:
- 3 replicas with automatic spreading across worker nodes
- Resource limits: 1 CPU / 512MB RAM per replica
- Resource reservations: 0.5 CPU / 256MB RAM per replica
- Rolling updates with automatic rollback
- Health checks every 30 seconds
- Traefik integration with automatic HTTPS
- Load balancer with health checking
- Docker Secrets integration for age keys
- Comprehensive logging configuration
**Environment Variables**:
- `LOG_LEVEL`: Logging verbosity
- `MCP_LOCAL`: MCP server URL (loopback only)
- `PORT`: HTTP server port (8443)
- `MAX_BODY_MB`: Request size limit
- `AGE_IDENT_PATH`: Age private key path
- `AGE_RECIPS_PATH`: Age public key path
- `KACHING_JWKS_URL`: KACHING JWKS endpoint
- `REQUIRED_SCOPE`: Required JWT scope
**Secrets**:
- `seqthink_age_identity`: Age private key (mounted at /run/secrets/)
- `seqthink_age_recipients`: Age public key (mounted at /run/secrets/)
**Network**:
- `chorus-overlay`: External overlay network for service mesh
**Labels**:
- Traefik routing: `seqthink.chorus.services`
- HTTPS with Let's Encrypt
- Health check path: `/health`
- Load balancer port: 8443
### 2. Deployment Documentation
**File**: `deploy/seqthink/DEPLOYMENT.md` (500+ lines)
**Sections**:
1. **Prerequisites**: Cluster setup, network requirements
2. **Architecture**: Security layers diagram
3. **Step-by-step deployment**:
- Generate age keys
- Create Docker secrets
- Build Docker image
- Deploy to swarm
- Verify deployment
- Test with JWT tokens
4. **Configuration reference**: All environment variables documented
5. **Scaling**: Horizontal scaling commands
6. **Updates**: Rolling update procedures
7. **Rollback**: Automatic and manual rollback
8. **Monitoring**: Prometheus metrics, health checks, logs
9. **Troubleshooting**: Common issues and solutions
10. **Security considerations**: Key rotation, TLS, rate limiting
11. **Development mode**: Testing without security
12. **Production checklist**: Pre-deployment verification
### 3. End-to-End Test Script
**File**: `deploy/seqthink/test-e2e.sh` (executable)
**Tests**:
1. Health endpoint validation
2. Readiness endpoint validation
3. Metrics endpoint verification
4. Unauthorized request rejection (401)
5. Invalid authorization header rejection (401)
6. JWT token validation (if token provided)
7. Encrypted request/response (if age keys provided)
8. Content-Type validation (415 for wrong type)
9. Metrics collection verification
10. SSE endpoint availability
**Usage**:
```bash
# Basic tests (no auth)
./deploy/seqthink/test-e2e.sh
# With JWT token
export JWT_TOKEN="eyJhbGci..."
./deploy/seqthink/test-e2e.sh
# With JWT + encryption
export JWT_TOKEN="eyJhbGci..."
export AGE_RECIPIENT="$(cat seqthink_age.pub)"
export AGE_IDENTITY="seqthink_age.key"
./deploy/seqthink/test-e2e.sh
```
**Output**:
- Color-coded test results (✓ pass, ✗ fail, ⚠ warn)
- Test summary with counts
- Exit code 0 if all pass, 1 if any fail
### 4. Secrets Management Guide
**File**: `deploy/seqthink/SECRETS.md` (400+ lines)
**Topics**:
1. **Secret types**: Age keys, KACHING config
2. **Key generation**:
- Method 1: Using age-keygen
- Method 2: Using Go code
3. **Storing secrets**: Docker Swarm secret commands
4. **Using secrets**: Compose file configuration
5. **Key rotation**:
- Why rotate
- 5-step rotation process
- Zero-downtime rotation
6. **Backup and recovery**:
- Secure backup procedures
- Age-encrypted backups
- Recovery process
7. **Security best practices**:
- Key generation ✓/✗ guidelines
- Key storage ✓/✗ guidelines
- Key distribution ✓/✗ guidelines
- Key lifecycle ✓/✗ guidelines
8. **Troubleshooting**: Common secret issues
9. **Client-side key management**: Distributing public keys
10. **Compliance and auditing**: SOC 2, ISO 27001
11. **Emergency procedures**: Key compromise response
## Deployment Flow
### Initial Deployment
```bash
# 1. Generate keys
age-keygen -o seqthink_age.key
age-keygen -y seqthink_age.key > seqthink_age.pub
# 2. Create secrets
docker secret create seqthink_age_identity seqthink_age.key
docker secret create seqthink_age_recipients seqthink_age.pub
# 3. Build image
docker build -f deploy/seqthink/Dockerfile -t anthonyrawlins/seqthink-wrapper:latest .
docker push anthonyrawlins/seqthink-wrapper:latest
# 4. Deploy
docker stack deploy -c deploy/seqthink/docker-compose.swarm.yml seqthink
# 5. Verify
docker service ps seqthink_seqthink-wrapper
docker service logs seqthink_seqthink-wrapper
# 6. Test
./deploy/seqthink/test-e2e.sh
```
### Update Deployment
```bash
# Build new version
docker build -f deploy/seqthink/Dockerfile -t anthonyrawlins/seqthink-wrapper:0.2.0 .
docker push anthonyrawlins/seqthink-wrapper:0.2.0
# Rolling update
docker service update \
--image anthonyrawlins/seqthink-wrapper:0.2.0 \
seqthink_seqthink-wrapper
```
## Service Architecture
```
Internet
↓
Traefik (HTTPS + Load Balancing)
↓
seqthink.chorus.services
↓
Docker Swarm Overlay Network
↓
SeqThink Wrapper (3 replicas)
├─ JWT Validation (KACHING)
├─ Age Decryption
├─ MCP Server (loopback)
├─ Age Encryption
└─ Response
```
## Security Layers
1. **Transport Security**: TLS 1.3 via Traefik
2. **Authentication**: JWT signature verification (RS256)
3. **Authorization**: Scope-based access control
4. **Encryption**: Age end-to-end encryption
5. **Network Isolation**: MCP server on loopback only
6. **Secrets Management**: Docker Swarm secrets (tmpfs)
7. **Resource Limits**: Container resource constraints
## Monitoring Integration
**Prometheus Metrics** (`/metrics`):
- `seqthink_requests_total`: Total requests
- `seqthink_errors_total`: Total errors
- `seqthink_decrypt_failures_total`: Decryption failures
- `seqthink_encrypt_failures_total`: Encryption failures
- `seqthink_policy_denials_total`: Authorization denials
- `seqthink_request_duration_seconds`: Request latency
**Health Checks**:
- `/health`: Liveness probe (wrapper running)
- `/ready`: Readiness probe (MCP server ready)
**Logging**:
- JSON format via docker logs
- 10MB max size, 3 file rotation
- Centralized log aggregation ready
## Production Readiness
✅ **High Availability**: 3 replicas with auto-restart
✅ **Zero-Downtime Updates**: Rolling updates with health checks
✅ **Automatic Rollback**: On update failure
✅ **Resource Management**: CPU/memory limits and reservations
✅ **Security**: Multi-layer defense (TLS, JWT, Age, Secrets)
✅ **Monitoring**: Metrics, health checks, structured logs
✅ **Documentation**: Complete deployment and operations guides
✅ **Testing**: Automated E2E test suite
✅ **Secrets Management**: Docker Swarm secrets with rotation procedures
## Next Steps
1. Test deployment on staging environment
2. Generate production age keys
3. Configure KACHING JWT integration
4. Deploy to production cluster
5. Monitor metrics and logs
6. Load testing and performance validation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
217 lines
6.4 KiB
Bash
Executable File
217 lines
6.4 KiB
Bash
Executable File
#!/bin/bash
|
|
# End-to-end test script for Sequential Thinking Age Wrapper
|
|
set -e
|
|
|
|
echo "🧪 Sequential Thinking Wrapper E2E Tests"
|
|
echo "========================================"
|
|
echo ""
|
|
|
|
# Configuration
|
|
WRAPPER_URL="${WRAPPER_URL:-http://localhost:8443}"
|
|
JWT_TOKEN="${JWT_TOKEN:-}"
|
|
AGE_RECIPIENT="${AGE_RECIPIENT:-}"
|
|
AGE_IDENTITY="${AGE_IDENTITY:-}"
|
|
|
|
# Color codes
|
|
GREEN='\033[0;32m'
|
|
RED='\033[0;31m'
|
|
YELLOW='\033[1;33m'
|
|
NC='\033[0m' # No Color
|
|
|
|
# Test counters
|
|
TESTS_RUN=0
|
|
TESTS_PASSED=0
|
|
TESTS_FAILED=0
|
|
|
|
# Helper functions
|
|
pass() {
|
|
echo -e "${GREEN}✓${NC} $1"
|
|
((TESTS_PASSED++))
|
|
}
|
|
|
|
fail() {
|
|
echo -e "${RED}✗${NC} $1"
|
|
((TESTS_FAILED++))
|
|
}
|
|
|
|
warn() {
|
|
echo -e "${YELLOW}⚠${NC} $1"
|
|
}
|
|
|
|
test_start() {
|
|
((TESTS_RUN++))
|
|
echo ""
|
|
echo "Test $TESTS_RUN: $1"
|
|
echo "---"
|
|
}
|
|
|
|
# Test 1: Health Check
|
|
test_start "Health endpoint"
|
|
if curl -sf "$WRAPPER_URL/health" > /dev/null 2>&1; then
|
|
pass "Health check passed"
|
|
else
|
|
fail "Health check failed"
|
|
fi
|
|
|
|
# Test 2: Readiness Check
|
|
test_start "Readiness endpoint"
|
|
if curl -sf "$WRAPPER_URL/ready" > /dev/null 2>&1; then
|
|
pass "Readiness check passed"
|
|
else
|
|
fail "Readiness check failed"
|
|
fi
|
|
|
|
# Test 3: Metrics Endpoint
|
|
test_start "Metrics endpoint"
|
|
if curl -sf "$WRAPPER_URL/metrics" | grep -q "seqthink_requests_total"; then
|
|
pass "Metrics endpoint accessible"
|
|
else
|
|
fail "Metrics endpoint failed"
|
|
fi
|
|
|
|
# Test 4: Unauthorized Request (no token)
|
|
test_start "Unauthorized request rejection"
|
|
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"tool":"test"}')
|
|
|
|
if [ "$HTTP_CODE" = "401" ]; then
|
|
pass "Unauthorized request correctly rejected (401)"
|
|
else
|
|
warn "Expected 401, got $HTTP_CODE (may be policy disabled)"
|
|
fi
|
|
|
|
# Test 5: Invalid Authorization Header
|
|
test_start "Invalid authorization header"
|
|
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \
|
|
-H "Authorization: InvalidFormat" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"tool":"test"}')
|
|
|
|
if [ "$HTTP_CODE" = "401" ]; then
|
|
pass "Invalid auth header correctly rejected (401)"
|
|
else
|
|
warn "Expected 401, got $HTTP_CODE (may be policy disabled)"
|
|
fi
|
|
|
|
# Test 6: JWT Token Validation (if token provided)
|
|
if [ -n "$JWT_TOKEN" ]; then
|
|
test_start "JWT token validation"
|
|
|
|
# Check if age keys are available
|
|
if [ -n "$AGE_RECIPIENT" ] && [ -n "$AGE_IDENTITY" ]; then
|
|
# Test with encryption
|
|
test_start "Encrypted request with valid JWT"
|
|
|
|
# Create test payload
|
|
TEST_PAYLOAD='{"tool":"mcp__sequential-thinking__sequentialthinking","payload":{"thought":"Test thought","thoughtNumber":1,"totalThoughts":1,"nextThoughtNeeded":false}}'
|
|
|
|
# Encrypt payload
|
|
ENCRYPTED_PAYLOAD=$(echo "$TEST_PAYLOAD" | age -r "$AGE_RECIPIENT" 2>/dev/null)
|
|
|
|
if [ $? -eq 0 ]; then
|
|
# Send encrypted request
|
|
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \
|
|
-H "Authorization: Bearer $JWT_TOKEN" \
|
|
-H "Content-Type: application/age" \
|
|
-d "$ENCRYPTED_PAYLOAD")
|
|
|
|
if [ "$HTTP_CODE" = "200" ]; then
|
|
pass "Encrypted request with JWT succeeded"
|
|
else
|
|
fail "Encrypted request failed with HTTP $HTTP_CODE"
|
|
fi
|
|
else
|
|
fail "Failed to encrypt test payload"
|
|
fi
|
|
else
|
|
# Test without encryption (plaintext mode)
|
|
test_start "Plaintext request with valid JWT"
|
|
|
|
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \
|
|
-H "Authorization: Bearer $JWT_TOKEN" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"tool":"mcp__sequential-thinking__sequentialthinking","payload":{"thought":"Test","thoughtNumber":1,"totalThoughts":1,"nextThoughtNeeded":false}}')
|
|
|
|
if [ "$HTTP_CODE" = "200" ]; then
|
|
pass "Plaintext request with JWT succeeded"
|
|
else
|
|
warn "Request failed with HTTP $HTTP_CODE"
|
|
fi
|
|
fi
|
|
else
|
|
warn "JWT_TOKEN not set - skipping authenticated tests"
|
|
fi
|
|
|
|
# Test 7: Content-Type Validation (if encryption enabled)
|
|
if [ -n "$AGE_RECIPIENT" ]; then
|
|
test_start "Content-Type validation for encrypted mode"
|
|
|
|
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \
|
|
-H "Authorization: Bearer ${JWT_TOKEN:-dummy}" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"tool":"test"}')
|
|
|
|
if [ "$HTTP_CODE" = "415" ]; then
|
|
pass "Incorrect Content-Type correctly rejected (415)"
|
|
else
|
|
warn "Expected 415, got $HTTP_CODE"
|
|
fi
|
|
fi
|
|
|
|
# Test 8: Metrics Collection
|
|
test_start "Metrics collection"
|
|
METRICS=$(curl -s "$WRAPPER_URL/metrics")
|
|
|
|
if echo "$METRICS" | grep -q "seqthink_requests_total"; then
|
|
REQUEST_COUNT=$(echo "$METRICS" | grep "^seqthink_requests_total" | awk '{print $2}')
|
|
pass "Request metrics collected (total: $REQUEST_COUNT)"
|
|
else
|
|
fail "Request metrics not found"
|
|
fi
|
|
|
|
if echo "$METRICS" | grep -q "seqthink_errors_total"; then
|
|
ERROR_COUNT=$(echo "$METRICS" | grep "^seqthink_errors_total" | awk '{print $2}')
|
|
pass "Error metrics collected (total: $ERROR_COUNT)"
|
|
else
|
|
fail "Error metrics not found"
|
|
fi
|
|
|
|
if echo "$METRICS" | grep -q "seqthink_policy_denials_total"; then
|
|
DENIAL_COUNT=$(echo "$METRICS" | grep "^seqthink_policy_denials_total" | awk '{print $2}')
|
|
pass "Policy denial metrics collected (total: $DENIAL_COUNT)"
|
|
else
|
|
warn "Policy denial metrics not found (may be policy disabled)"
|
|
fi
|
|
|
|
# Test 9: SSE Endpoint (basic check)
|
|
test_start "SSE endpoint availability"
|
|
# Just check if endpoint exists, don't try to consume stream
|
|
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 2 "$WRAPPER_URL/mcp/sse" 2>/dev/null || echo "timeout")
|
|
|
|
if [ "$HTTP_CODE" = "401" ] || [ "$HTTP_CODE" = "200" ]; then
|
|
pass "SSE endpoint exists (HTTP $HTTP_CODE)"
|
|
else
|
|
warn "SSE endpoint check inconclusive (HTTP $HTTP_CODE)"
|
|
fi
|
|
|
|
# Summary
|
|
echo ""
|
|
echo "========================================"
|
|
echo "Test Summary"
|
|
echo "========================================"
|
|
echo "Tests Run: $TESTS_RUN"
|
|
echo -e "${GREEN}Tests Passed: $TESTS_PASSED${NC}"
|
|
if [ $TESTS_FAILED -gt 0 ]; then
|
|
echo -e "${RED}Tests Failed: $TESTS_FAILED${NC}"
|
|
fi
|
|
echo ""
|
|
|
|
if [ $TESTS_FAILED -eq 0 ]; then
|
|
echo -e "${GREEN}✓ All tests passed!${NC}"
|
|
exit 0
|
|
else
|
|
echo -e "${RED}✗ Some tests failed${NC}"
|
|
exit 1
|
|
fi
|