CHORUS/deploy/seqthink/SECRETS.md

Secrets Management Guide

Overview

The Sequential Thinking Wrapper uses Docker Secrets for secure key management. This guide covers generating, storing, and rotating secrets.

Secret Types

1. Age Encryption Keys

Purpose: End-to-end encryption of MCP communications

Components:

• Identity (Private Key): seqthink_age_identity
• Recipients (Public Key): seqthink_age_recipients

2. KACHING JWT Configuration

Purpose: Authentication and authorization

Components:

• JWKS URL (environment variable, not a secret)
• Required scope (environment variable, not a secret)

Generating Age Keys

Method 1: Using age-keygen

# Install age if not already installed
# macOS: brew install age
# Ubuntu: apt install age
# Arch: pacman -S age

# Generate identity (private key)
age-keygen -o seqthink_age.key

# Extract recipient (public key)
age-keygen -y seqthink_age.key > seqthink_age.pub

Output Format:

seqthink_age.key:

# created: 2025-10-13T08:00:00+11:00
# public key: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
AGE-SECRET-KEY-1GFPYYSJQ...

seqthink_age.pub:

age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

Method 2: Using Go Code

Create a helper script:

package main

import (
	"filippo.io/age"
	"fmt"
	"os"
)

func main() {
	identity, err := age.GenerateX25519Identity()
	if err != nil {
		panic(err)
	}

	// Write identity (private key)
	identityFile, _ := os.Create("seqthink_age.key")
	fmt.Fprintf(identityFile, "# created: %s\n", time.Now().Format(time.RFC3339))
	fmt.Fprintf(identityFile, "# public key: %s\n", identity.Recipient().String())
	fmt.Fprintf(identityFile, "%s\n", identity.String())
	identityFile.Close()

	// Write recipient (public key)
	recipientFile, _ := os.Create("seqthink_age.pub")
	fmt.Fprintf(recipientFile, "%s\n", identity.Recipient().String())
	recipientFile.Close()

	fmt.Println("✅ Keys generated:")
	fmt.Println("  Identity:  seqthink_age.key")
	fmt.Println("  Recipient: seqthink_age.pub")
}

Storing Secrets in Docker Swarm

Create Secrets

# Create identity secret
docker secret create seqthink_age_identity seqthink_age.key

# Create recipient secret
docker secret create seqthink_age_recipients seqthink_age.pub

Verify Secrets

# List secrets
docker secret ls | grep seqthink

# Inspect secret metadata (not content)
docker secret inspect seqthink_age_identity

Expected Output:

[
    {
        "ID": "abc123...",
        "Version": {
            "Index": 123
        },
        "CreatedAt": "2025-10-13T08:00:00.000Z",
        "UpdatedAt": "2025-10-13T08:00:00.000Z",
        "Spec": {
            "Name": "seqthink_age_identity",
            "Labels": {}
        }
    }
]

Using Secrets in Services

Compose File Configuration

services:
  seqthink-wrapper:
    environment:
      AGE_IDENT_PATH: /run/secrets/seqthink_age_identity
      AGE_RECIPS_PATH: /run/secrets/seqthink_age_recipients

    secrets:
      - seqthink_age_identity
      - seqthink_age_recipients

secrets:
  seqthink_age_identity:
    external: true
  seqthink_age_recipients:
    external: true

Secret Mount Points

Inside the container, secrets are available at:

• /run/secrets/seqthink_age_identity
• /run/secrets/seqthink_age_recipients

These are read-only files mounted via tmpfs.

Key Rotation

Why Rotate Keys?

• Compromised key material
• Compliance requirements
• Periodic security hygiene
• Employee offboarding

Rotation Process

Step 1: Generate New Keys

# Generate new keys with timestamp
TIMESTAMP=$(date +%Y%m%d)
age-keygen -o seqthink_age_${TIMESTAMP}.key
age-keygen -y seqthink_age_${TIMESTAMP}.key > seqthink_age_${TIMESTAMP}.pub

Step 2: Create New Secrets

# Create new secrets with version suffix
docker secret create seqthink_age_identity_v2 seqthink_age_${TIMESTAMP}.key
docker secret create seqthink_age_recipients_v2 seqthink_age_${TIMESTAMP}.pub

Step 3: Update Service

# Update service to use new secrets
docker service update \
  --secret-rm seqthink_age_identity \
  --secret-add source=seqthink_age_identity_v2,target=seqthink_age_identity \
  --secret-rm seqthink_age_recipients \
  --secret-add source=seqthink_age_recipients_v2,target=seqthink_age_recipients \
  seqthink_seqthink-wrapper

Step 4: Verify New Keys Work

# Check service logs
docker service logs seqthink_seqthink-wrapper | tail -20

# Test encryption with new keys
echo "test" | age -r "$(cat seqthink_age_${TIMESTAMP}.pub)" | \
  age -d -i seqthink_age_${TIMESTAMP}.key

Step 5: Clean Up Old Secrets

# Wait 24 hours to ensure no rollback needed
# Then remove old secrets
docker secret rm seqthink_age_identity
docker secret rm seqthink_age_recipients

# Promote v2 to primary names (optional)
docker secret create seqthink_age_identity seqthink_age_${TIMESTAMP}.key
docker secret create seqthink_age_recipients seqthink_age_${TIMESTAMP}.pub

Backup and Recovery

Backup Keys

# Create secure backup directory
mkdir -p ~/secure-backups/seqthink-keys
chmod 700 ~/secure-backups/seqthink-keys

# Copy keys to backup
cp seqthink_age.key ~/secure-backups/seqthink-keys/
cp seqthink_age.pub ~/secure-backups/seqthink-keys/

# Encrypt backup
tar czf - ~/secure-backups/seqthink-keys | \
  age -r age1... > seqthink-keys-backup.tar.gz.age

# Store encrypted backup in:
# 1. Offsite backup (Backblaze, Scaleway)
# 2. Password manager (1Password, Bitwarden)
# 3. Hardware security module (YubiKey)

Recover Keys

# Decrypt backup
age -d -i master_identity.key seqthink-keys-backup.tar.gz.age | \
  tar xzf -

# Recreate Docker secrets
docker secret create seqthink_age_identity \
  ~/secure-backups/seqthink-keys/seqthink_age.key
docker secret create seqthink_age_recipients \
  ~/secure-backups/seqthink-keys/seqthink_age.pub

Security Best Practices

1. Key Generation

✅ DO:

• Generate keys on secure, air-gapped machines
• Use cryptographically secure random number generators
• Generate new keys per environment (dev, staging, prod)

❌ DON'T:

• Reuse keys across environments
• Generate keys on shared/untrusted systems
• Store keys in git repositories

2. Key Storage

✅ DO:

• Use Docker Secrets for production
• Encrypt backups with age or GPG
• Store backups in multiple secure locations
• Use hardware security modules for highly sensitive keys

❌ DON'T:

• Store keys in environment variables
• Commit keys to version control
• Share keys via insecure channels (email, Slack)
• Store unencrypted keys on disk

3. Key Distribution

✅ DO:

• Use secure channels (age-encrypted files, password managers)
• Verify key fingerprints before use
• Use Docker Secrets for service access
• Document key distribution recipients

❌ DON'T:

• Send keys via unencrypted email
• Post keys in chat systems
• Share keys verbally
• Use public key servers for private keys

4. Key Lifecycle

✅ DO:

• Rotate keys periodically (quarterly recommended)
• Rotate keys immediately if compromised
• Keep audit log of key generations and rotations
• Test key recovery procedures

❌ DON'T:

• Keep keys indefinitely without rotation
• Delete old keys immediately (keep 30-day overlap)
• Skip testing key recovery
• Forget to document key changes

Troubleshooting

Issue: Secret Not Found

Error:

Error response from daemon: secret 'seqthink_age_identity' not found

Solution:

# Check if secret exists
docker secret ls | grep seqthink

# If missing, create it
docker secret create seqthink_age_identity seqthink_age.key

Issue: Permission Denied Reading Secret

Error:

open /run/secrets/seqthink_age_identity: permission denied

Solution:

• Secrets are mounted read-only to containers
• Container user must have read permissions
• Check Dockerfile USER directive

Issue: Wrong Key Used

Error:

Failed to decrypt request
seqthink_decrypt_failures_total increasing

Solution:

# Verify public key matches private key
PUBLIC_FROM_PRIVATE=$(age-keygen -y seqthink_age.key)
PUBLIC_IN_SECRET=$(cat seqthink_age.pub)

if [ "$PUBLIC_FROM_PRIVATE" = "$PUBLIC_IN_SECRET" ]; then
  echo "✓ Keys match"
else
  echo "✗ Keys don't match - regenerate recipient"
fi

Issue: Secret Update Not Taking Effect

Symptoms: Service still using old keys after update

Solution:

# Force service update
docker service update --force seqthink_seqthink-wrapper

# Or restart service
docker service scale seqthink_seqthink-wrapper=0
docker service scale seqthink_seqthink-wrapper=3

Client-Side Key Management

Distributing Public Keys to Clients

Clients need the public key to encrypt requests:

# Generate client-friendly recipient file
cat seqthink_age.pub

# Clients can encrypt with:
echo '{"tool":"test","payload":{}}' | age -r age1ql3z7hjy54pw3... > request.age

Recipient Key Distribution Methods

1. Configuration Management:

   seqthink:
     recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
   

2. Environment Variable:

   export SEQTHINK_RECIPIENT="age1ql3z7hjy54pw3..."
   

3. API Discovery (future):

   curl https://seqthink.chorus.services/.well-known/age-recipient
   

Compliance and Auditing

Audit Log Example

Maintain a log of key operations:

# seqthink-keys-audit.md

## 2025-10-13 - Initial Key Generation
- Generated by: Tony
- Purpose: Production deployment
- Public key: age1ql3z7hjy54pw3...
- Stored in: Docker Secrets + Backup

## 2025-11-15 - Quarterly Rotation
- Generated by: Tony
- Reason: Scheduled quarterly rotation
- Old public key: age1ql3z7hjy54pw3...
- New public key: age1abc123xyz...
- Overlap period: 30 days
- Old keys removed: 2025-12-15

Compliance Requirements

For SOC 2, ISO 27001, or similar:

• Document key generation procedures
• Log all key rotations
• Restrict key access to authorized personnel
• Encrypt keys at rest
• Regular key rotation (90 days recommended)
• Incident response plan for key compromise

Emergency Procedures

Key Compromise Response

If keys are compromised:

1. Immediate Actions (< 1 hour):

   # Generate new keys immediately
   age-keygen -o seqthink_age_emergency.key
   age-keygen -y seqthink_age_emergency.key > seqthink_age_emergency.pub
   
   # Update Docker secrets
   docker secret create seqthink_age_identity_emergency seqthink_age_emergency.key
   docker secret create seqthink_age_recipients_emergency seqthink_age_emergency.pub
   
   # Force service update
   docker service update --force \
     --secret-rm seqthink_age_identity \
     --secret-add source=seqthink_age_identity_emergency,target=seqthink_age_identity \
     --secret-rm seqthink_age_recipients \
     --secret-add source=seqthink_age_recipients_emergency,target=seqthink_age_recipients \
     seqthink_seqthink-wrapper
   

2. Communication (< 4 hours):

   • Notify all clients of new public key
   • Update documentation
   • Post mortem analysis

3. Follow-up (< 24 hours):

   • Review access logs
   • Identify compromise source
   • Update security procedures
   • Complete incident report

References

• age encryption tool
• Docker Secrets documentation
• NIST Key Management Guidelines