# Secrets Management Guide ## Overview The Sequential Thinking Wrapper uses Docker Secrets for secure key management. This guide covers generating, storing, and rotating secrets. ## Secret Types ### 1. Age Encryption Keys **Purpose**: End-to-end encryption of MCP communications **Components**: - **Identity (Private Key)**: `seqthink_age_identity` - **Recipients (Public Key)**: `seqthink_age_recipients` ### 2. KACHING JWT Configuration **Purpose**: Authentication and authorization **Components**: - JWKS URL (environment variable, not a secret) - Required scope (environment variable, not a secret) ## Generating Age Keys ### Method 1: Using age-keygen ```bash # Install age if not already installed # macOS: brew install age # Ubuntu: apt install age # Arch: pacman -S age # Generate identity (private key) age-keygen -o seqthink_age.key # Extract recipient (public key) age-keygen -y seqthink_age.key > seqthink_age.pub ``` **Output Format**: `seqthink_age.key`: ``` # created: 2025-10-13T08:00:00+11:00 # public key: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p AGE-SECRET-KEY-1GFPYYSJQ... ``` `seqthink_age.pub`: ``` age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p ``` ### Method 2: Using Go Code Create a helper script: ```go package main import ( "filippo.io/age" "fmt" "os" ) func main() { identity, err := age.GenerateX25519Identity() if err != nil { panic(err) } // Write identity (private key) identityFile, _ := os.Create("seqthink_age.key") fmt.Fprintf(identityFile, "# created: %s\n", time.Now().Format(time.RFC3339)) fmt.Fprintf(identityFile, "# public key: %s\n", identity.Recipient().String()) fmt.Fprintf(identityFile, "%s\n", identity.String()) identityFile.Close() // Write recipient (public key) recipientFile, _ := os.Create("seqthink_age.pub") fmt.Fprintf(recipientFile, "%s\n", identity.Recipient().String()) recipientFile.Close() fmt.Println("✅ Keys generated:") fmt.Println(" Identity: seqthink_age.key") fmt.Println(" Recipient: seqthink_age.pub") } ``` ## Storing Secrets in Docker Swarm ### Create Secrets ```bash # Create identity secret docker secret create seqthink_age_identity seqthink_age.key # Create recipient secret docker secret create seqthink_age_recipients seqthink_age.pub ``` ### Verify Secrets ```bash # List secrets docker secret ls | grep seqthink # Inspect secret metadata (not content) docker secret inspect seqthink_age_identity ``` **Expected Output**: ```json [ { "ID": "abc123...", "Version": { "Index": 123 }, "CreatedAt": "2025-10-13T08:00:00.000Z", "UpdatedAt": "2025-10-13T08:00:00.000Z", "Spec": { "Name": "seqthink_age_identity", "Labels": {} } } ] ``` ## Using Secrets in Services ### Compose File Configuration ```yaml services: seqthink-wrapper: environment: AGE_IDENT_PATH: /run/secrets/seqthink_age_identity AGE_RECIPS_PATH: /run/secrets/seqthink_age_recipients secrets: - seqthink_age_identity - seqthink_age_recipients secrets: seqthink_age_identity: external: true seqthink_age_recipients: external: true ``` ### Secret Mount Points Inside the container, secrets are available at: - `/run/secrets/seqthink_age_identity` - `/run/secrets/seqthink_age_recipients` These are read-only files mounted via tmpfs. ## Key Rotation ### Why Rotate Keys? - Compromised key material - Compliance requirements - Periodic security hygiene - Employee offboarding ### Rotation Process #### Step 1: Generate New Keys ```bash # Generate new keys with timestamp TIMESTAMP=$(date +%Y%m%d) age-keygen -o seqthink_age_${TIMESTAMP}.key age-keygen -y seqthink_age_${TIMESTAMP}.key > seqthink_age_${TIMESTAMP}.pub ``` #### Step 2: Create New Secrets ```bash # Create new secrets with version suffix docker secret create seqthink_age_identity_v2 seqthink_age_${TIMESTAMP}.key docker secret create seqthink_age_recipients_v2 seqthink_age_${TIMESTAMP}.pub ``` #### Step 3: Update Service ```bash # Update service to use new secrets docker service update \ --secret-rm seqthink_age_identity \ --secret-add source=seqthink_age_identity_v2,target=seqthink_age_identity \ --secret-rm seqthink_age_recipients \ --secret-add source=seqthink_age_recipients_v2,target=seqthink_age_recipients \ seqthink_seqthink-wrapper ``` #### Step 4: Verify New Keys Work ```bash # Check service logs docker service logs seqthink_seqthink-wrapper | tail -20 # Test encryption with new keys echo "test" | age -r "$(cat seqthink_age_${TIMESTAMP}.pub)" | \ age -d -i seqthink_age_${TIMESTAMP}.key ``` #### Step 5: Clean Up Old Secrets ```bash # Wait 24 hours to ensure no rollback needed # Then remove old secrets docker secret rm seqthink_age_identity docker secret rm seqthink_age_recipients # Promote v2 to primary names (optional) docker secret create seqthink_age_identity seqthink_age_${TIMESTAMP}.key docker secret create seqthink_age_recipients seqthink_age_${TIMESTAMP}.pub ``` ## Backup and Recovery ### Backup Keys ```bash # Create secure backup directory mkdir -p ~/secure-backups/seqthink-keys chmod 700 ~/secure-backups/seqthink-keys # Copy keys to backup cp seqthink_age.key ~/secure-backups/seqthink-keys/ cp seqthink_age.pub ~/secure-backups/seqthink-keys/ # Encrypt backup tar czf - ~/secure-backups/seqthink-keys | \ age -r age1... > seqthink-keys-backup.tar.gz.age # Store encrypted backup in: # 1. Offsite backup (Backblaze, Scaleway) # 2. Password manager (1Password, Bitwarden) # 3. Hardware security module (YubiKey) ``` ### Recover Keys ```bash # Decrypt backup age -d -i master_identity.key seqthink-keys-backup.tar.gz.age | \ tar xzf - # Recreate Docker secrets docker secret create seqthink_age_identity \ ~/secure-backups/seqthink-keys/seqthink_age.key docker secret create seqthink_age_recipients \ ~/secure-backups/seqthink-keys/seqthink_age.pub ``` ## Security Best Practices ### 1. Key Generation ✅ **DO**: - Generate keys on secure, air-gapped machines - Use cryptographically secure random number generators - Generate new keys per environment (dev, staging, prod) ❌ **DON'T**: - Reuse keys across environments - Generate keys on shared/untrusted systems - Store keys in git repositories ### 2. Key Storage ✅ **DO**: - Use Docker Secrets for production - Encrypt backups with age or GPG - Store backups in multiple secure locations - Use hardware security modules for highly sensitive keys ❌ **DON'T**: - Store keys in environment variables - Commit keys to version control - Share keys via insecure channels (email, Slack) - Store unencrypted keys on disk ### 3. Key Distribution ✅ **DO**: - Use secure channels (age-encrypted files, password managers) - Verify key fingerprints before use - Use Docker Secrets for service access - Document key distribution recipients ❌ **DON'T**: - Send keys via unencrypted email - Post keys in chat systems - Share keys verbally - Use public key servers for private keys ### 4. Key Lifecycle ✅ **DO**: - Rotate keys periodically (quarterly recommended) - Rotate keys immediately if compromised - Keep audit log of key generations and rotations - Test key recovery procedures ❌ **DON'T**: - Keep keys indefinitely without rotation - Delete old keys immediately (keep 30-day overlap) - Skip testing key recovery - Forget to document key changes ## Troubleshooting ### Issue: Secret Not Found **Error**: ``` Error response from daemon: secret 'seqthink_age_identity' not found ``` **Solution**: ```bash # Check if secret exists docker secret ls | grep seqthink # If missing, create it docker secret create seqthink_age_identity seqthink_age.key ``` ### Issue: Permission Denied Reading Secret **Error**: ``` open /run/secrets/seqthink_age_identity: permission denied ``` **Solution**: - Secrets are mounted read-only to containers - Container user must have read permissions - Check Dockerfile USER directive ### Issue: Wrong Key Used **Error**: ``` Failed to decrypt request seqthink_decrypt_failures_total increasing ``` **Solution**: ```bash # Verify public key matches private key PUBLIC_FROM_PRIVATE=$(age-keygen -y seqthink_age.key) PUBLIC_IN_SECRET=$(cat seqthink_age.pub) if [ "$PUBLIC_FROM_PRIVATE" = "$PUBLIC_IN_SECRET" ]; then echo "✓ Keys match" else echo "✗ Keys don't match - regenerate recipient" fi ``` ### Issue: Secret Update Not Taking Effect **Symptoms**: Service still using old keys after update **Solution**: ```bash # Force service update docker service update --force seqthink_seqthink-wrapper # Or restart service docker service scale seqthink_seqthink-wrapper=0 docker service scale seqthink_seqthink-wrapper=3 ``` ## Client-Side Key Management ### Distributing Public Keys to Clients Clients need the public key to encrypt requests: ```bash # Generate client-friendly recipient file cat seqthink_age.pub # Clients can encrypt with: echo '{"tool":"test","payload":{}}' | age -r age1ql3z7hjy54pw3... > request.age ``` ### Recipient Key Distribution Methods 1. **Configuration Management**: ```yaml seqthink: recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p ``` 2. **Environment Variable**: ```bash export SEQTHINK_RECIPIENT="age1ql3z7hjy54pw3..." ``` 3. **API Discovery** (future): ```bash curl https://seqthink.chorus.services/.well-known/age-recipient ``` ## Compliance and Auditing ### Audit Log Example Maintain a log of key operations: ```markdown # seqthink-keys-audit.md ## 2025-10-13 - Initial Key Generation - Generated by: Tony - Purpose: Production deployment - Public key: age1ql3z7hjy54pw3... - Stored in: Docker Secrets + Backup ## 2025-11-15 - Quarterly Rotation - Generated by: Tony - Reason: Scheduled quarterly rotation - Old public key: age1ql3z7hjy54pw3... - New public key: age1abc123xyz... - Overlap period: 30 days - Old keys removed: 2025-12-15 ``` ### Compliance Requirements For SOC 2, ISO 27001, or similar: - Document key generation procedures - Log all key rotations - Restrict key access to authorized personnel - Encrypt keys at rest - Regular key rotation (90 days recommended) - Incident response plan for key compromise ## Emergency Procedures ### Key Compromise Response If keys are compromised: 1. **Immediate Actions** (< 1 hour): ```bash # Generate new keys immediately age-keygen -o seqthink_age_emergency.key age-keygen -y seqthink_age_emergency.key > seqthink_age_emergency.pub # Update Docker secrets docker secret create seqthink_age_identity_emergency seqthink_age_emergency.key docker secret create seqthink_age_recipients_emergency seqthink_age_emergency.pub # Force service update docker service update --force \ --secret-rm seqthink_age_identity \ --secret-add source=seqthink_age_identity_emergency,target=seqthink_age_identity \ --secret-rm seqthink_age_recipients \ --secret-add source=seqthink_age_recipients_emergency,target=seqthink_age_recipients \ seqthink_seqthink-wrapper ``` 2. **Communication** (< 4 hours): - Notify all clients of new public key - Update documentation - Post mortem analysis 3. **Follow-up** (< 24 hours): - Review access logs - Identify compromise source - Update security procedures - Complete incident report ## References - [age encryption tool](https://github.com/FiloSottile/age) - [Docker Secrets documentation](https://docs.docker.com/engine/swarm/secrets/) - [NIST Key Management Guidelines](https://csrc.nist.gov/publications/detail/sp/800-57-part-1/rev-5/final)