diff --git a/deploy/seqthink/DEPLOYMENT.md b/deploy/seqthink/DEPLOYMENT.md new file mode 100644 index 0000000..cfb9526 --- /dev/null +++ b/deploy/seqthink/DEPLOYMENT.md @@ -0,0 +1,380 @@ +# Sequential Thinking Age Wrapper - Deployment Guide + +## Overview + +This guide covers deploying the Sequential Thinking Age-Encrypted Wrapper to Docker Swarm with full security enabled. + +## Prerequisites + +- Docker Swarm cluster initialized +- `chorus-overlay` network created +- Traefik reverse proxy configured +- KACHING authentication service available + +## Architecture + +``` +Client โ†’ Traefik (HTTPS) โ†’ SeqThink Wrapper (JWT + Age Encryption) โ†’ MCP Server (loopback) +``` + +**Security Layers**: +1. **TLS**: Traefik terminates HTTPS +2. **JWT**: KACHING token validation +3. **Age Encryption**: End-to-end encrypted payloads + +## Step 1: Generate Age Keys + +Generate a key pair for encryption: + +```bash +# Generate age identity (private key) +age-keygen -o seqthink_age.key + +# Extract recipient (public key) +age-keygen -y seqthink_age.key > seqthink_age.pub +``` + +**Output**: +``` +seqthink_age.key: +# created: 2025-10-13T08:00:00+11:00 +# public key: age1abcd... +AGE-SECRET-KEY-1ABCD... + +seqthink_age.pub: +age1abcd... +``` + +## Step 2: Create Docker Secrets + +Store the age keys as Docker secrets: + +```bash +# Create identity secret +docker secret create seqthink_age_identity seqthink_age.key + +# Create recipient secret +docker secret create seqthink_age_recipients seqthink_age.pub + +# Verify secrets +docker secret ls | grep seqthink +``` + +**Expected Output**: +``` +seqthink_age_identity +seqthink_age_recipients +``` + +## Step 3: Build Docker Image + +Build the wrapper image: + +```bash +cd /home/tony/chorus/project-queues/active/CHORUS + +# Build image +docker build -f deploy/seqthink/Dockerfile -t anthonyrawlins/seqthink-wrapper:latest . + +# Tag with version +docker tag anthonyrawlins/seqthink-wrapper:latest anthonyrawlins/seqthink-wrapper:0.1.0 + +# Push to registry +docker push anthonyrawlins/seqthink-wrapper:latest +docker push anthonyrawlins/seqthink-wrapper:0.1.0 +``` + +## Step 4: Deploy to Swarm + +Deploy the service: + +```bash +cd deploy/seqthink + +# Deploy stack +docker stack deploy -c docker-compose.swarm.yml seqthink + +# Check service status +docker service ls | grep seqthink + +# Check logs +docker service logs -f seqthink_seqthink-wrapper +``` + +**Expected Log Output**: +``` +๐Ÿš€ Starting Sequential Thinking Age Wrapper +โณ Waiting for MCP server... +โœ… MCP server ready +Policy enforcement enabled + jwks_url: https://auth.kaching.services/jwks + required_scope: sequentialthinking.run +Fetching JWKS +JWKS cached successfully + key_count: 2 +Encryption enabled - using encrypted endpoint +๐Ÿ” Wrapper listening + addr: :8443 + encryption_enabled: true + policy_enabled: true +``` + +## Step 5: Verify Deployment + +Check service health: + +```bash +# Check replicas +docker service ps seqthink_seqthink-wrapper + +# Test health endpoint +curl -f http://localhost:8443/health +# Expected: OK + +# Test readiness +curl -f http://localhost:8443/ready +# Expected: READY + +# Check metrics +curl http://localhost:8443/metrics | grep seqthink +``` + +## Step 6: Test with JWT Token + +Get a KACHING JWT token and test the API: + +```bash +# Set your JWT token +export JWT_TOKEN="eyJhbGciOiJSUzI1NiIsImtpZCI6ImRlZmF1bHQiLCJ0eXAiOiJKV1QifQ..." + +# Test unauthorized (should fail) +curl -X POST https://seqthink.chorus.services/mcp/tool \ + -H "Content-Type: application/age" \ + -d "test" +# Expected: 401 Unauthorized + +# Test authorized (should succeed) +curl -X POST https://seqthink.chorus.services/mcp/tool \ + -H "Authorization: Bearer $JWT_TOKEN" \ + -H "Content-Type: application/age" \ + -d "$(echo '{"tool":"test","payload":{}}' | age -r $(cat seqthink_age.pub))" \ + --output encrypted_response.age + +# Decrypt response +age -d -i seqthink_age.key encrypted_response.age +``` + +## Configuration Reference + +### Environment Variables + +| Variable | Required | Default | Description | +|----------|----------|---------|-------------| +| `PORT` | No | `8443` | HTTP server port | +| `MCP_LOCAL` | No | `http://127.0.0.1:8000` | MCP server URL (loopback) | +| `LOG_LEVEL` | No | `info` | Logging level (debug, info, warn, error) | +| `MAX_BODY_MB` | No | `4` | Maximum request body size in MB | +| `AGE_IDENT_PATH` | **Yes** | - | Path to age identity (private key) | +| `AGE_RECIPS_PATH` | **Yes** | - | Path to age recipients (public key) | +| `KACHING_JWKS_URL` | **Yes** | - | KACHING JWKS endpoint | +| `REQUIRED_SCOPE` | **Yes** | `sequentialthinking.run` | Required JWT scope | + +### Docker Secrets + +| Secret Name | Purpose | Content | +|-------------|---------|---------| +| `seqthink_age_identity` | Age private key | `AGE-SECRET-KEY-1...` | +| `seqthink_age_recipients` | Age public key | `age1...` | + +### Network Ports + +| Port | Protocol | Purpose | +|------|----------|---------| +| `8443` | HTTP | Wrapper API | +| `8000` | HTTP | MCP server (internal loopback only) | + +## Scaling + +Scale the service: + +```bash +# Scale up +docker service scale seqthink_seqthink-wrapper=5 + +# Scale down +docker service scale seqthink_seqthink-wrapper=2 +``` + +## Updates + +Rolling update: + +```bash +# Build new version +docker build -f deploy/seqthink/Dockerfile -t anthonyrawlins/seqthink-wrapper:0.2.0 . +docker push anthonyrawlins/seqthink-wrapper:0.2.0 + +# Update service +docker service update \ + --image anthonyrawlins/seqthink-wrapper:0.2.0 \ + seqthink_seqthink-wrapper + +# Monitor rollout +docker service ps seqthink_seqthink-wrapper +``` + +## Rollback + +If update fails: + +```bash +# Automatic rollback (configured in stack) +# Or manual rollback: +docker service rollback seqthink_seqthink-wrapper +``` + +## Monitoring + +### Prometheus Metrics + +Available at `http://localhost:8443/metrics`: + +``` +seqthink_requests_total +seqthink_errors_total +seqthink_decrypt_failures_total +seqthink_encrypt_failures_total +seqthink_policy_denials_total +seqthink_request_duration_seconds +``` + +### Health Checks + +- **Liveness**: `GET /health` - Returns 200 if wrapper is running +- **Readiness**: `GET /ready` - Returns 200 if MCP server is ready + +### Logs + +View logs: + +```bash +# All replicas +docker service logs seqthink_seqthink-wrapper + +# Follow logs +docker service logs -f seqthink_seqthink-wrapper + +# Specific replica +docker service logs seqthink_seqthink-wrapper. +``` + +## Troubleshooting + +### Issue: Policy Enforcement Disabled + +**Symptoms**: +``` +Policy enforcement disabled - no JWKS URL or required scope configured +``` + +**Solution**: +- Verify `KACHING_JWKS_URL` and `REQUIRED_SCOPE` are set +- Check environment variables: `docker service inspect seqthink_seqthink-wrapper` + +### Issue: JWKS Fetch Failed + +**Symptoms**: +``` +Failed to pre-fetch JWKS, will retry on first request +``` + +**Solution**: +- Check KACHING service is accessible +- Verify JWKS URL is correct +- Check network connectivity + +### Issue: Decryption Failed + +**Symptoms**: +``` +Failed to decrypt request +seqthink_decrypt_failures_total increasing +``` + +**Solution**: +- Verify age keys match between client and server +- Check client is using correct public key +- Ensure secrets are correctly mounted + +### Issue: MCP Server Not Ready + +**Symptoms**: +``` +โŒ MCP server not ready +timeout waiting for MCP server +``` + +**Solution**: +- Check MCP server is starting correctly +- Review entrypoint.sh logs +- Verify Python dependencies installed + +## Security Considerations + +1. **Key Rotation**: Periodically rotate age keys: + ```bash + # Generate new keys + age-keygen -o seqthink_age_new.key + age-keygen -y seqthink_age_new.key > seqthink_age_new.pub + + # Update secrets (requires service restart) + docker secret rm seqthink_age_identity + docker secret create seqthink_age_identity seqthink_age_new.key + ``` + +2. **JWT Token Expiration**: Tokens should have short expiration times (1 hour recommended) + +3. **Network Isolation**: MCP server only accessible on loopback (127.0.0.1) + +4. **TLS**: Always use HTTPS in production (via Traefik) + +5. **Rate Limiting**: Consider adding rate limiting at Traefik level + +## Development Mode + +For testing without security: + +```yaml +environment: + # Disable encryption + AGE_IDENT_PATH: "" + AGE_RECIPS_PATH: "" + + # Disable policy + KACHING_JWKS_URL: "" + REQUIRED_SCOPE: "" +``` + +**WARNING**: Only use in development environments! + +## Production Checklist + +- [ ] Age keys generated and stored as Docker secrets +- [ ] KACHING JWKS URL configured and accessible +- [ ] Docker image built and pushed to registry +- [ ] Service deployed to swarm +- [ ] Health checks passing +- [ ] Metrics endpoint accessible +- [ ] JWT tokens validated successfully +- [ ] End-to-end encryption verified +- [ ] Logs show no errors +- [ ] Monitoring alerts configured +- [ ] Backup of age keys stored securely +- [ ] Documentation updated with deployment details + +## Support + +For issues or questions: +- Check logs: `docker service logs seqthink_seqthink-wrapper` +- Review metrics: `curl http://localhost:8443/metrics` +- Consult implementation docs in `/home/tony/chorus/project-queues/active/CHORUS/docs/` diff --git a/deploy/seqthink/SECRETS.md b/deploy/seqthink/SECRETS.md new file mode 100644 index 0000000..a431b55 --- /dev/null +++ b/deploy/seqthink/SECRETS.md @@ -0,0 +1,491 @@ +# Secrets Management Guide + +## Overview + +The Sequential Thinking Wrapper uses Docker Secrets for secure key management. This guide covers generating, storing, and rotating secrets. + +## Secret Types + +### 1. Age Encryption Keys + +**Purpose**: End-to-end encryption of MCP communications + +**Components**: +- **Identity (Private Key)**: `seqthink_age_identity` +- **Recipients (Public Key)**: `seqthink_age_recipients` + +### 2. KACHING JWT Configuration + +**Purpose**: Authentication and authorization + +**Components**: +- JWKS URL (environment variable, not a secret) +- Required scope (environment variable, not a secret) + +## Generating Age Keys + +### Method 1: Using age-keygen + +```bash +# Install age if not already installed +# macOS: brew install age +# Ubuntu: apt install age +# Arch: pacman -S age + +# Generate identity (private key) +age-keygen -o seqthink_age.key + +# Extract recipient (public key) +age-keygen -y seqthink_age.key > seqthink_age.pub +``` + +**Output Format**: + +`seqthink_age.key`: +``` +# created: 2025-10-13T08:00:00+11:00 +# public key: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p +AGE-SECRET-KEY-1GFPYYSJQ... +``` + +`seqthink_age.pub`: +``` +age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p +``` + +### Method 2: Using Go Code + +Create a helper script: + +```go +package main + +import ( + "filippo.io/age" + "fmt" + "os" +) + +func main() { + identity, err := age.GenerateX25519Identity() + if err != nil { + panic(err) + } + + // Write identity (private key) + identityFile, _ := os.Create("seqthink_age.key") + fmt.Fprintf(identityFile, "# created: %s\n", time.Now().Format(time.RFC3339)) + fmt.Fprintf(identityFile, "# public key: %s\n", identity.Recipient().String()) + fmt.Fprintf(identityFile, "%s\n", identity.String()) + identityFile.Close() + + // Write recipient (public key) + recipientFile, _ := os.Create("seqthink_age.pub") + fmt.Fprintf(recipientFile, "%s\n", identity.Recipient().String()) + recipientFile.Close() + + fmt.Println("โœ… Keys generated:") + fmt.Println(" Identity: seqthink_age.key") + fmt.Println(" Recipient: seqthink_age.pub") +} +``` + +## Storing Secrets in Docker Swarm + +### Create Secrets + +```bash +# Create identity secret +docker secret create seqthink_age_identity seqthink_age.key + +# Create recipient secret +docker secret create seqthink_age_recipients seqthink_age.pub +``` + +### Verify Secrets + +```bash +# List secrets +docker secret ls | grep seqthink + +# Inspect secret metadata (not content) +docker secret inspect seqthink_age_identity +``` + +**Expected Output**: +```json +[ + { + "ID": "abc123...", + "Version": { + "Index": 123 + }, + "CreatedAt": "2025-10-13T08:00:00.000Z", + "UpdatedAt": "2025-10-13T08:00:00.000Z", + "Spec": { + "Name": "seqthink_age_identity", + "Labels": {} + } + } +] +``` + +## Using Secrets in Services + +### Compose File Configuration + +```yaml +services: + seqthink-wrapper: + environment: + AGE_IDENT_PATH: /run/secrets/seqthink_age_identity + AGE_RECIPS_PATH: /run/secrets/seqthink_age_recipients + + secrets: + - seqthink_age_identity + - seqthink_age_recipients + +secrets: + seqthink_age_identity: + external: true + seqthink_age_recipients: + external: true +``` + +### Secret Mount Points + +Inside the container, secrets are available at: +- `/run/secrets/seqthink_age_identity` +- `/run/secrets/seqthink_age_recipients` + +These are read-only files mounted via tmpfs. + +## Key Rotation + +### Why Rotate Keys? + +- Compromised key material +- Compliance requirements +- Periodic security hygiene +- Employee offboarding + +### Rotation Process + +#### Step 1: Generate New Keys + +```bash +# Generate new keys with timestamp +TIMESTAMP=$(date +%Y%m%d) +age-keygen -o seqthink_age_${TIMESTAMP}.key +age-keygen -y seqthink_age_${TIMESTAMP}.key > seqthink_age_${TIMESTAMP}.pub +``` + +#### Step 2: Create New Secrets + +```bash +# Create new secrets with version suffix +docker secret create seqthink_age_identity_v2 seqthink_age_${TIMESTAMP}.key +docker secret create seqthink_age_recipients_v2 seqthink_age_${TIMESTAMP}.pub +``` + +#### Step 3: Update Service + +```bash +# Update service to use new secrets +docker service update \ + --secret-rm seqthink_age_identity \ + --secret-add source=seqthink_age_identity_v2,target=seqthink_age_identity \ + --secret-rm seqthink_age_recipients \ + --secret-add source=seqthink_age_recipients_v2,target=seqthink_age_recipients \ + seqthink_seqthink-wrapper +``` + +#### Step 4: Verify New Keys Work + +```bash +# Check service logs +docker service logs seqthink_seqthink-wrapper | tail -20 + +# Test encryption with new keys +echo "test" | age -r "$(cat seqthink_age_${TIMESTAMP}.pub)" | \ + age -d -i seqthink_age_${TIMESTAMP}.key +``` + +#### Step 5: Clean Up Old Secrets + +```bash +# Wait 24 hours to ensure no rollback needed +# Then remove old secrets +docker secret rm seqthink_age_identity +docker secret rm seqthink_age_recipients + +# Promote v2 to primary names (optional) +docker secret create seqthink_age_identity seqthink_age_${TIMESTAMP}.key +docker secret create seqthink_age_recipients seqthink_age_${TIMESTAMP}.pub +``` + +## Backup and Recovery + +### Backup Keys + +```bash +# Create secure backup directory +mkdir -p ~/secure-backups/seqthink-keys +chmod 700 ~/secure-backups/seqthink-keys + +# Copy keys to backup +cp seqthink_age.key ~/secure-backups/seqthink-keys/ +cp seqthink_age.pub ~/secure-backups/seqthink-keys/ + +# Encrypt backup +tar czf - ~/secure-backups/seqthink-keys | \ + age -r age1... > seqthink-keys-backup.tar.gz.age + +# Store encrypted backup in: +# 1. Offsite backup (Backblaze, Scaleway) +# 2. Password manager (1Password, Bitwarden) +# 3. Hardware security module (YubiKey) +``` + +### Recover Keys + +```bash +# Decrypt backup +age -d -i master_identity.key seqthink-keys-backup.tar.gz.age | \ + tar xzf - + +# Recreate Docker secrets +docker secret create seqthink_age_identity \ + ~/secure-backups/seqthink-keys/seqthink_age.key +docker secret create seqthink_age_recipients \ + ~/secure-backups/seqthink-keys/seqthink_age.pub +``` + +## Security Best Practices + +### 1. Key Generation + +โœ… **DO**: +- Generate keys on secure, air-gapped machines +- Use cryptographically secure random number generators +- Generate new keys per environment (dev, staging, prod) + +โŒ **DON'T**: +- Reuse keys across environments +- Generate keys on shared/untrusted systems +- Store keys in git repositories + +### 2. Key Storage + +โœ… **DO**: +- Use Docker Secrets for production +- Encrypt backups with age or GPG +- Store backups in multiple secure locations +- Use hardware security modules for highly sensitive keys + +โŒ **DON'T**: +- Store keys in environment variables +- Commit keys to version control +- Share keys via insecure channels (email, Slack) +- Store unencrypted keys on disk + +### 3. Key Distribution + +โœ… **DO**: +- Use secure channels (age-encrypted files, password managers) +- Verify key fingerprints before use +- Use Docker Secrets for service access +- Document key distribution recipients + +โŒ **DON'T**: +- Send keys via unencrypted email +- Post keys in chat systems +- Share keys verbally +- Use public key servers for private keys + +### 4. Key Lifecycle + +โœ… **DO**: +- Rotate keys periodically (quarterly recommended) +- Rotate keys immediately if compromised +- Keep audit log of key generations and rotations +- Test key recovery procedures + +โŒ **DON'T**: +- Keep keys indefinitely without rotation +- Delete old keys immediately (keep 30-day overlap) +- Skip testing key recovery +- Forget to document key changes + +## Troubleshooting + +### Issue: Secret Not Found + +**Error**: +``` +Error response from daemon: secret 'seqthink_age_identity' not found +``` + +**Solution**: +```bash +# Check if secret exists +docker secret ls | grep seqthink + +# If missing, create it +docker secret create seqthink_age_identity seqthink_age.key +``` + +### Issue: Permission Denied Reading Secret + +**Error**: +``` +open /run/secrets/seqthink_age_identity: permission denied +``` + +**Solution**: +- Secrets are mounted read-only to containers +- Container user must have read permissions +- Check Dockerfile USER directive + +### Issue: Wrong Key Used + +**Error**: +``` +Failed to decrypt request +seqthink_decrypt_failures_total increasing +``` + +**Solution**: +```bash +# Verify public key matches private key +PUBLIC_FROM_PRIVATE=$(age-keygen -y seqthink_age.key) +PUBLIC_IN_SECRET=$(cat seqthink_age.pub) + +if [ "$PUBLIC_FROM_PRIVATE" = "$PUBLIC_IN_SECRET" ]; then + echo "โœ“ Keys match" +else + echo "โœ— Keys don't match - regenerate recipient" +fi +``` + +### Issue: Secret Update Not Taking Effect + +**Symptoms**: Service still using old keys after update + +**Solution**: +```bash +# Force service update +docker service update --force seqthink_seqthink-wrapper + +# Or restart service +docker service scale seqthink_seqthink-wrapper=0 +docker service scale seqthink_seqthink-wrapper=3 +``` + +## Client-Side Key Management + +### Distributing Public Keys to Clients + +Clients need the public key to encrypt requests: + +```bash +# Generate client-friendly recipient file +cat seqthink_age.pub + +# Clients can encrypt with: +echo '{"tool":"test","payload":{}}' | age -r age1ql3z7hjy54pw3... > request.age +``` + +### Recipient Key Distribution Methods + +1. **Configuration Management**: + ```yaml + seqthink: + recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p + ``` + +2. **Environment Variable**: + ```bash + export SEQTHINK_RECIPIENT="age1ql3z7hjy54pw3..." + ``` + +3. **API Discovery** (future): + ```bash + curl https://seqthink.chorus.services/.well-known/age-recipient + ``` + +## Compliance and Auditing + +### Audit Log Example + +Maintain a log of key operations: + +```markdown +# seqthink-keys-audit.md + +## 2025-10-13 - Initial Key Generation +- Generated by: Tony +- Purpose: Production deployment +- Public key: age1ql3z7hjy54pw3... +- Stored in: Docker Secrets + Backup + +## 2025-11-15 - Quarterly Rotation +- Generated by: Tony +- Reason: Scheduled quarterly rotation +- Old public key: age1ql3z7hjy54pw3... +- New public key: age1abc123xyz... +- Overlap period: 30 days +- Old keys removed: 2025-12-15 +``` + +### Compliance Requirements + +For SOC 2, ISO 27001, or similar: +- Document key generation procedures +- Log all key rotations +- Restrict key access to authorized personnel +- Encrypt keys at rest +- Regular key rotation (90 days recommended) +- Incident response plan for key compromise + +## Emergency Procedures + +### Key Compromise Response + +If keys are compromised: + +1. **Immediate Actions** (< 1 hour): + ```bash + # Generate new keys immediately + age-keygen -o seqthink_age_emergency.key + age-keygen -y seqthink_age_emergency.key > seqthink_age_emergency.pub + + # Update Docker secrets + docker secret create seqthink_age_identity_emergency seqthink_age_emergency.key + docker secret create seqthink_age_recipients_emergency seqthink_age_emergency.pub + + # Force service update + docker service update --force \ + --secret-rm seqthink_age_identity \ + --secret-add source=seqthink_age_identity_emergency,target=seqthink_age_identity \ + --secret-rm seqthink_age_recipients \ + --secret-add source=seqthink_age_recipients_emergency,target=seqthink_age_recipients \ + seqthink_seqthink-wrapper + ``` + +2. **Communication** (< 4 hours): + - Notify all clients of new public key + - Update documentation + - Post mortem analysis + +3. **Follow-up** (< 24 hours): + - Review access logs + - Identify compromise source + - Update security procedures + - Complete incident report + +## References + +- [age encryption tool](https://github.com/FiloSottile/age) +- [Docker Secrets documentation](https://docs.docker.com/engine/swarm/secrets/) +- [NIST Key Management Guidelines](https://csrc.nist.gov/publications/detail/sp/800-57-part-1/rev-5/final) diff --git a/deploy/seqthink/docker-compose.swarm.yml b/deploy/seqthink/docker-compose.swarm.yml new file mode 100644 index 0000000..13d94bd --- /dev/null +++ b/deploy/seqthink/docker-compose.swarm.yml @@ -0,0 +1,102 @@ +version: '3.8' + +services: + seqthink-wrapper: + image: anthonyrawlins/seqthink-wrapper:latest + networks: + - chorus-overlay + ports: + - "8443:8443" + environment: + # Logging + LOG_LEVEL: info + + # MCP server (internal loopback) + MCP_LOCAL: http://127.0.0.1:8000 + + # Port configuration + PORT: "8443" + + # Request limits + MAX_BODY_MB: "4" + + # Age encryption (use secrets) + AGE_IDENT_PATH: /run/secrets/seqthink_age_identity + AGE_RECIPS_PATH: /run/secrets/seqthink_age_recipients + + # KACHING JWT policy + KACHING_JWKS_URL: https://auth.kaching.services/jwks + REQUIRED_SCOPE: sequentialthinking.run + + secrets: + - seqthink_age_identity + - seqthink_age_recipients + + deploy: + mode: replicated + replicas: 3 + placement: + constraints: + - node.role == worker + preferences: + - spread: node.hostname + + resources: + limits: + cpus: '1.0' + memory: 512M + reservations: + cpus: '0.5' + memory: 256M + + restart_policy: + condition: on-failure + delay: 5s + max_attempts: 3 + window: 120s + + update_config: + parallelism: 1 + delay: 10s + failure_action: rollback + monitor: 30s + max_failure_ratio: 0.3 + + rollback_config: + parallelism: 1 + delay: 5s + failure_action: pause + monitor: 30s + + labels: + - "traefik.enable=true" + - "traefik.http.routers.seqthink.rule=Host(`seqthink.chorus.services`)" + - "traefik.http.routers.seqthink.entrypoints=websecure" + - "traefik.http.routers.seqthink.tls=true" + - "traefik.http.routers.seqthink.tls.certresolver=letsencrypt" + - "traefik.http.services.seqthink.loadbalancer.server.port=8443" + - "traefik.http.services.seqthink.loadbalancer.healthcheck.path=/health" + - "traefik.http.services.seqthink.loadbalancer.healthcheck.interval=30s" + + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8443/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 40s + + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + +networks: + chorus-overlay: + external: true + +secrets: + seqthink_age_identity: + external: true + seqthink_age_recipients: + external: true diff --git a/deploy/seqthink/test-e2e.sh b/deploy/seqthink/test-e2e.sh new file mode 100755 index 0000000..050e9fe --- /dev/null +++ b/deploy/seqthink/test-e2e.sh @@ -0,0 +1,216 @@ +#!/bin/bash +# End-to-end test script for Sequential Thinking Age Wrapper +set -e + +echo "๐Ÿงช Sequential Thinking Wrapper E2E Tests" +echo "========================================" +echo "" + +# Configuration +WRAPPER_URL="${WRAPPER_URL:-http://localhost:8443}" +JWT_TOKEN="${JWT_TOKEN:-}" +AGE_RECIPIENT="${AGE_RECIPIENT:-}" +AGE_IDENTITY="${AGE_IDENTITY:-}" + +# Color codes +GREEN='\033[0;32m' +RED='\033[0;31m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +# Test counters +TESTS_RUN=0 +TESTS_PASSED=0 +TESTS_FAILED=0 + +# Helper functions +pass() { + echo -e "${GREEN}โœ“${NC} $1" + ((TESTS_PASSED++)) +} + +fail() { + echo -e "${RED}โœ—${NC} $1" + ((TESTS_FAILED++)) +} + +warn() { + echo -e "${YELLOW}โš ${NC} $1" +} + +test_start() { + ((TESTS_RUN++)) + echo "" + echo "Test $TESTS_RUN: $1" + echo "---" +} + +# Test 1: Health Check +test_start "Health endpoint" +if curl -sf "$WRAPPER_URL/health" > /dev/null 2>&1; then + pass "Health check passed" +else + fail "Health check failed" +fi + +# Test 2: Readiness Check +test_start "Readiness endpoint" +if curl -sf "$WRAPPER_URL/ready" > /dev/null 2>&1; then + pass "Readiness check passed" +else + fail "Readiness check failed" +fi + +# Test 3: Metrics Endpoint +test_start "Metrics endpoint" +if curl -sf "$WRAPPER_URL/metrics" | grep -q "seqthink_requests_total"; then + pass "Metrics endpoint accessible" +else + fail "Metrics endpoint failed" +fi + +# Test 4: Unauthorized Request (no token) +test_start "Unauthorized request rejection" +HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \ + -H "Content-Type: application/json" \ + -d '{"tool":"test"}') + +if [ "$HTTP_CODE" = "401" ]; then + pass "Unauthorized request correctly rejected (401)" +else + warn "Expected 401, got $HTTP_CODE (may be policy disabled)" +fi + +# Test 5: Invalid Authorization Header +test_start "Invalid authorization header" +HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \ + -H "Authorization: InvalidFormat" \ + -H "Content-Type: application/json" \ + -d '{"tool":"test"}') + +if [ "$HTTP_CODE" = "401" ]; then + pass "Invalid auth header correctly rejected (401)" +else + warn "Expected 401, got $HTTP_CODE (may be policy disabled)" +fi + +# Test 6: JWT Token Validation (if token provided) +if [ -n "$JWT_TOKEN" ]; then + test_start "JWT token validation" + + # Check if age keys are available + if [ -n "$AGE_RECIPIENT" ] && [ -n "$AGE_IDENTITY" ]; then + # Test with encryption + test_start "Encrypted request with valid JWT" + + # Create test payload + TEST_PAYLOAD='{"tool":"mcp__sequential-thinking__sequentialthinking","payload":{"thought":"Test thought","thoughtNumber":1,"totalThoughts":1,"nextThoughtNeeded":false}}' + + # Encrypt payload + ENCRYPTED_PAYLOAD=$(echo "$TEST_PAYLOAD" | age -r "$AGE_RECIPIENT" 2>/dev/null) + + if [ $? -eq 0 ]; then + # Send encrypted request + HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \ + -H "Authorization: Bearer $JWT_TOKEN" \ + -H "Content-Type: application/age" \ + -d "$ENCRYPTED_PAYLOAD") + + if [ "$HTTP_CODE" = "200" ]; then + pass "Encrypted request with JWT succeeded" + else + fail "Encrypted request failed with HTTP $HTTP_CODE" + fi + else + fail "Failed to encrypt test payload" + fi + else + # Test without encryption (plaintext mode) + test_start "Plaintext request with valid JWT" + + HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \ + -H "Authorization: Bearer $JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"tool":"mcp__sequential-thinking__sequentialthinking","payload":{"thought":"Test","thoughtNumber":1,"totalThoughts":1,"nextThoughtNeeded":false}}') + + if [ "$HTTP_CODE" = "200" ]; then + pass "Plaintext request with JWT succeeded" + else + warn "Request failed with HTTP $HTTP_CODE" + fi + fi +else + warn "JWT_TOKEN not set - skipping authenticated tests" +fi + +# Test 7: Content-Type Validation (if encryption enabled) +if [ -n "$AGE_RECIPIENT" ]; then + test_start "Content-Type validation for encrypted mode" + + HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$WRAPPER_URL/mcp/tool" \ + -H "Authorization: Bearer ${JWT_TOKEN:-dummy}" \ + -H "Content-Type: application/json" \ + -d '{"tool":"test"}') + + if [ "$HTTP_CODE" = "415" ]; then + pass "Incorrect Content-Type correctly rejected (415)" + else + warn "Expected 415, got $HTTP_CODE" + fi +fi + +# Test 8: Metrics Collection +test_start "Metrics collection" +METRICS=$(curl -s "$WRAPPER_URL/metrics") + +if echo "$METRICS" | grep -q "seqthink_requests_total"; then + REQUEST_COUNT=$(echo "$METRICS" | grep "^seqthink_requests_total" | awk '{print $2}') + pass "Request metrics collected (total: $REQUEST_COUNT)" +else + fail "Request metrics not found" +fi + +if echo "$METRICS" | grep -q "seqthink_errors_total"; then + ERROR_COUNT=$(echo "$METRICS" | grep "^seqthink_errors_total" | awk '{print $2}') + pass "Error metrics collected (total: $ERROR_COUNT)" +else + fail "Error metrics not found" +fi + +if echo "$METRICS" | grep -q "seqthink_policy_denials_total"; then + DENIAL_COUNT=$(echo "$METRICS" | grep "^seqthink_policy_denials_total" | awk '{print $2}') + pass "Policy denial metrics collected (total: $DENIAL_COUNT)" +else + warn "Policy denial metrics not found (may be policy disabled)" +fi + +# Test 9: SSE Endpoint (basic check) +test_start "SSE endpoint availability" +# Just check if endpoint exists, don't try to consume stream +HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 2 "$WRAPPER_URL/mcp/sse" 2>/dev/null || echo "timeout") + +if [ "$HTTP_CODE" = "401" ] || [ "$HTTP_CODE" = "200" ]; then + pass "SSE endpoint exists (HTTP $HTTP_CODE)" +else + warn "SSE endpoint check inconclusive (HTTP $HTTP_CODE)" +fi + +# Summary +echo "" +echo "========================================" +echo "Test Summary" +echo "========================================" +echo "Tests Run: $TESTS_RUN" +echo -e "${GREEN}Tests Passed: $TESTS_PASSED${NC}" +if [ $TESTS_FAILED -gt 0 ]; then + echo -e "${RED}Tests Failed: $TESTS_FAILED${NC}" +fi +echo "" + +if [ $TESTS_FAILED -eq 0 ]; then + echo -e "${GREEN}โœ“ All tests passed!${NC}" + exit 0 +else + echo -e "${RED}โœ— Some tests failed${NC}" + exit 1 +fi