docs: Update README and add comprehensive CONFIGURATION guide

## Documentation Updates ### README.md - Production Status Update - Changed status from "MVP → Production Ready Transition" to "PRODUCTION READY ✅" - Added comprehensive Council Formation workflow (7-step process) - Updated architecture components with security stack - Enhanced API reference with authentication requirements - Added production deployment instructions - Comprehensive security section with enterprise-grade features - OpenTelemetry tracing and observability documentation - Updated development roadmap with phase completion status ### CONFIGURATION.md - New Comprehensive Guide - Complete reference for 60+ environment variables - Categorized sections: Database, Security, External Services, Feature Flags - Production and development configuration templates - Security best practices and hardening recommendations - Validation guide with common errors and troubleshooting - Performance tuning recommendations ## Key Highlights - Production-ready status clearly communicated - All new security features documented - Complete configuration management guide - Enterprise deployment procedures - Comprehensive observability setup 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-12 22:55:27 +10:00
parent 131868bdca
commit e5555ae277
2 changed files with 816 additions and 109 deletions
--- a/docs/CONFIGURATION.md
+++ b/docs/CONFIGURATION.md
@@ -0,0 +1,459 @@
+# WHOOSH Configuration Guide
+
+This guide provides comprehensive documentation for all WHOOSH configuration options and environment variables.
+
+## 📋 Quick Reference
+
+| Category | Variables | Description |
+|----------|-----------|-------------|
+| [Database](#database-configuration) | `WHOOSH_DATABASE_*` | PostgreSQL connection and pooling |
+| [Gitea Integration](#gitea-integration) | `WHOOSH_GITEA_*` | Repository monitoring and webhooks |
+| [Security](#security-configuration) | `WHOOSH_JWT_*`, `WHOOSH_CORS_*` | Authentication and access control |
+| [External Services](#external-services) | `WHOOSH_N8N_*`, `WHOOSH_BACKBEAT_*` | Third-party integrations |
+| [Feature Flags](#feature-flags) | `WHOOSH_FEATURE_*` | Optional functionality toggles |
+| [Docker Integration](#docker-integration) | `WHOOSH_DOCKER_*` | Container orchestration |
+| [Observability](#observability-configuration) | `WHOOSH_OTEL_*`, `WHOOSH_LOG_*` | Tracing and logging |
+
+## 🗄️ Database Configuration
+
+### Core Database Settings
+
+```bash
+# Primary database connection
+WHOOSH_DATABASE_URL=postgres://username:password@host:5432/database?sslmode=require
+
+# Alternative: Individual components
+WHOOSH_DB_HOST=localhost
+WHOOSH_DB_PORT=5432
+WHOOSH_DB_NAME=whoosh
+WHOOSH_DB_USER=whoosh_user
+WHOOSH_DB_PASSWORD=secure_password
+WHOOSH_DB_SSLMODE=require
+```
+
+### Connection Pool Settings
+
+```bash
+# Connection pool configuration
+WHOOSH_DB_MAX_OPEN_CONNS=25          # Maximum open connections
+WHOOSH_DB_MAX_IDLE_CONNS=10          # Maximum idle connections
+WHOOSH_DB_CONN_MAX_LIFETIME=300s     # Connection lifetime
+WHOOSH_DB_CONN_MAX_IDLE_TIME=60s     # Maximum idle time
+```
+
+### Migration Settings
+
+```bash
+# Database migration configuration
+WHOOSH_DB_MIGRATE_ON_START=true      # Run migrations on startup
+WHOOSH_MIGRATION_PATH=./migrations   # Migration files location
+```
+
+## 🔧 Gitea Integration
+
+### Basic Gitea Settings
+
+```bash
+# Gitea instance configuration
+WHOOSH_GITEA_URL=https://gitea.example.com
+WHOOSH_GITEA_TOKEN_FILE=/run/secrets/gitea_token    # Recommended for production
+WHOOSH_GITEA_TOKEN=your-gitea-api-token             # Alternative for development
+
+# Webhook configuration
+WHOOSH_WEBHOOK_SECRET_FILE=/run/secrets/webhook_secret
+WHOOSH_WEBHOOK_SECRET=your-webhook-secret
+```
+
+### Repository Monitoring
+
+```bash
+# Repository sync behavior
+WHOOSH_GITEA_EAGER_FILTER=true       # API-level filtering (recommended)
+WHOOSH_GITEA_FULL_RESCAN=false       # Complete vs incremental scan
+WHOOSH_GITEA_DEBUG_URLS=false        # Log exact API URLs for debugging
+
+# Retry and timeout settings
+WHOOSH_GITEA_MAX_RETRIES=3           # API retry attempts
+WHOOSH_GITEA_RETRY_DELAY=2s          # Delay between retries
+WHOOSH_GITEA_REQUEST_TIMEOUT=30s     # API request timeout
+```
+
+### Label and Issue Configuration
+
+```bash
+# Label management
+WHOOSH_CHORUS_TASK_LABELS=chorus-entrypoint,bzzz-task
+WHOOSH_AUTO_CREATE_LABELS=true       # Auto-create missing labels
+WHOOSH_ENABLE_CHORUS_INTEGRATION=true
+
+# Issue processing
+WHOOSH_ISSUE_BATCH_SIZE=50           # Issues per API request
+WHOOSH_ISSUE_SYNC_INTERVAL=300s      # Sync frequency
+```
+
+## 🔐 Security Configuration
+
+### JWT Authentication
+
+```bash
+# JWT token configuration
+WHOOSH_JWT_SECRET_FILE=/run/secrets/jwt_secret      # Recommended
+WHOOSH_JWT_SECRET=your-jwt-secret                   # Alternative
+WHOOSH_JWT_EXPIRATION=24h            # Token expiration
+WHOOSH_JWT_ISSUER=whoosh             # Token issuer
+WHOOSH_JWT_ALGORITHM=HS256           # Signing algorithm
+```
+
+### CORS Settings
+
+```bash
+# CORS configuration - NEVER use * in production
+WHOOSH_CORS_ALLOWED_ORIGINS=https://app.example.com,https://admin.example.com
+WHOOSH_CORS_ALLOWED_METHODS=GET,POST,PUT,DELETE,OPTIONS
+WHOOSH_CORS_ALLOWED_HEADERS=Authorization,Content-Type,X-Requested-With
+WHOOSH_CORS_ALLOW_CREDENTIALS=true
+WHOOSH_CORS_MAX_AGE=86400            # Preflight cache duration
+```
+
+### Rate Limiting
+
+```bash
+# Rate limiting configuration
+WHOOSH_RATE_LIMIT_ENABLED=true
+WHOOSH_RATE_LIMIT_REQUESTS=100       # Requests per window
+WHOOSH_RATE_LIMIT_WINDOW=60s         # Rate limiting window
+WHOOSH_RATE_LIMIT_CLEANUP_INTERVAL=300s  # Cleanup frequency
+```
+
+### Input Validation
+
+```bash
+# Request validation settings
+WHOOSH_MAX_REQUEST_SIZE=1048576      # 1MB default request size
+WHOOSH_MAX_WEBHOOK_SIZE=10485760     # 10MB for webhooks
+WHOOSH_VALIDATION_STRICT=true        # Enable strict validation
+```
+
+### Service Tokens
+
+```bash
+# Service-to-service authentication
+WHOOSH_SERVICE_TOKEN_FILE=/run/secrets/service_token
+WHOOSH_SERVICE_TOKEN=your-service-token
+WHOOSH_SERVICE_TOKEN_HEADER=X-Service-Token
+```
+
+## 🔗 External Services
+
+### N8N Integration
+
+```bash
+# N8N workflow automation
+WHOOSH_N8N_BASE_URL=https://n8n.example.com
+WHOOSH_N8N_AUTH_TOKEN_FILE=/run/secrets/n8n_token
+WHOOSH_N8N_AUTH_TOKEN=your-n8n-token
+WHOOSH_N8N_TIMEOUT=60s               # Request timeout
+WHOOSH_N8N_MAX_RETRIES=3             # Retry attempts
+```
+
+### BackBeat Monitoring
+
+```bash
+# BackBeat performance monitoring
+WHOOSH_BACKBEAT_URL=http://backbeat:3001
+WHOOSH_BACKBEAT_ENABLED=true
+WHOOSH_BACKBEAT_TOKEN_FILE=/run/secrets/backbeat_token
+WHOOSH_BACKBEAT_BEAT_INTERVAL=30s    # Beat frequency
+WHOOSH_BACKBEAT_TIMEOUT=10s          # Request timeout
+```
+
+## 🚩 Feature Flags
+
+### LLM Integration
+
+```bash
+# AI vs Heuristic classification
+WHOOSH_FEATURE_LLM_CLASSIFICATION=false     # Enable LLM classification
+WHOOSH_FEATURE_LLM_SKILL_ANALYSIS=false     # Enable LLM skill analysis
+WHOOSH_FEATURE_LLM_TEAM_MATCHING=false      # Enable LLM team matching
+WHOOSH_FEATURE_ENABLE_ANALYSIS_LOGGING=true # Log analysis details
+WHOOSH_FEATURE_ENABLE_FAILSAFE_FALLBACK=true # Fallback to heuristics
+```
+
+### Experimental Features
+
+```bash
+# Advanced features (use with caution)
+WHOOSH_FEATURE_ADVANCED_P2P=false           # Enhanced P2P discovery
+WHOOSH_FEATURE_CROSS_COUNCIL_COORDINATION=false
+WHOOSH_FEATURE_PREDICTIVE_FORMATION=false   # ML-based team formation
+WHOOSH_FEATURE_AUTO_SCALING=false           # Automatic agent scaling
+```
+
+## 🐳 Docker Integration
+
+### Docker Swarm Settings
+
+```bash
+# Docker daemon connection
+WHOOSH_DOCKER_ENABLED=true
+WHOOSH_DOCKER_HOST=unix:///var/run/docker.sock
+WHOOSH_DOCKER_VERSION=1.41           # Docker API version
+WHOOSH_DOCKER_TIMEOUT=60s            # Operation timeout
+
+# Swarm-specific settings
+WHOOSH_SWARM_NETWORK=chorus_default  # Swarm network name
+WHOOSH_SWARM_CONSTRAINTS=node.role==worker  # Placement constraints
+```
+
+### Agent Deployment
+
+```bash
+# CHORUS agent deployment
+WHOOSH_AGENT_IMAGE=anthonyrawlins/chorus:latest
+WHOOSH_AGENT_MEMORY_LIMIT=2048m      # Memory limit per agent
+WHOOSH_AGENT_CPU_LIMIT=1.0           # CPU limit per agent
+WHOOSH_AGENT_RESTART_POLICY=on-failure
+WHOOSH_AGENT_MAX_RESTARTS=3
+```
+
+### Volume and Secret Mounts
+
+```bash
+# Shared volumes
+WHOOSH_PROMPTS_PATH=/rust/containers/WHOOSH/prompts
+WHOOSH_SHARED_DATA_PATH=/rust/shared
+
+# Docker secrets
+WHOOSH_DOCKER_SECRET_PREFIX=whoosh_  # Secret naming prefix
+```
+
+## 📊 Observability Configuration
+
+### OpenTelemetry Tracing
+
+```bash
+# OpenTelemetry configuration
+WHOOSH_OTEL_ENABLED=true
+WHOOSH_OTEL_SERVICE_NAME=whoosh
+WHOOSH_OTEL_SERVICE_VERSION=1.0.0
+WHOOSH_OTEL_ENDPOINT=http://jaeger:14268/api/traces
+WHOOSH_OTEL_SAMPLER_RATIO=1.0        # Sampling ratio (0.0-1.0)
+WHOOSH_OTEL_BATCH_TIMEOUT=5s         # Batch export timeout
+```
+
+### Logging Configuration
+
+```bash
+# Logging settings
+WHOOSH_LOG_LEVEL=info                # trace, debug, info, warn, error
+WHOOSH_LOG_FORMAT=json               # json or text
+WHOOSH_LOG_OUTPUT=stdout             # stdout, stderr, or file path
+WHOOSH_LOG_CALLER=false              # Include caller information
+WHOOSH_LOG_TIMESTAMP=true            # Include timestamps
+```
+
+### Metrics and Health
+
+```bash
+# Prometheus metrics
+WHOOSH_METRICS_ENABLED=true
+WHOOSH_METRICS_PATH=/metrics         # Metrics endpoint path
+WHOOSH_METRICS_NAMESPACE=whoosh      # Metrics namespace
+
+# Health check configuration
+WHOOSH_HEALTH_CHECK_INTERVAL=30s     # Internal health check frequency
+WHOOSH_HEALTH_TIMEOUT=10s            # Health check timeout
+```
+
+## 🌐 Server Configuration
+
+### HTTP Server Settings
+
+```bash
+# Server bind configuration
+WHOOSH_SERVER_HOST=0.0.0.0           # Bind address
+WHOOSH_SERVER_PORT=8080              # Listen port
+WHOOSH_SERVER_READ_TIMEOUT=30s       # Request read timeout
+WHOOSH_SERVER_WRITE_TIMEOUT=30s      # Response write timeout
+WHOOSH_SERVER_IDLE_TIMEOUT=60s       # Idle connection timeout
+WHOOSH_SERVER_MAX_HEADER_BYTES=1048576 # Max header size
+```
+
+### TLS Configuration
+
+```bash
+# TLS/SSL settings (optional)
+WHOOSH_TLS_ENABLED=false
+WHOOSH_TLS_CERT_FILE=/path/to/cert.pem
+WHOOSH_TLS_KEY_FILE=/path/to/key.pem
+WHOOSH_TLS_MIN_VERSION=1.2           # Minimum TLS version
+```
+
+## 🔍 P2P Discovery Configuration
+
+### Service Discovery
+
+```bash
+# P2P discovery settings
+WHOOSH_P2P_DISCOVERY_ENABLED=true
+WHOOSH_P2P_KNOWN_ENDPOINTS=chorus:8081,agent1:8081,agent2:8081
+WHOOSH_P2P_SERVICE_PORTS=8081,8082,8083
+WHOOSH_P2P_DOCKER_ENABLED=true       # Docker Swarm discovery
+
+# Health checking
+WHOOSH_P2P_HEALTH_TIMEOUT=5s         # Agent health check timeout
+WHOOSH_P2P_RETRY_ATTEMPTS=3          # Health check retries
+WHOOSH_P2P_DISCOVERY_INTERVAL=60s    # Discovery cycle frequency
+```
+
+### Agent Filtering
+
+```bash
+# Agent capability filtering
+WHOOSH_P2P_REQUIRED_CAPABILITIES=council,reasoning
+WHOOSH_P2P_MIN_AGENT_VERSION=1.0.0   # Minimum agent version
+WHOOSH_P2P_FILTER_INACTIVE=true      # Filter inactive agents
+```
+
+## 📁 Environment File Examples
+
+### Production Environment (.env.production)
+
+```bash
+# Production configuration template
+# Copy to .env and customize
+
+# Database
+WHOOSH_DATABASE_URL=postgres://whoosh:${DB_PASSWORD}@postgres:5432/whoosh?sslmode=require
+WHOOSH_DB_MAX_OPEN_CONNS=50
+WHOOSH_DB_MAX_IDLE_CONNS=20
+
+# Security (use Docker secrets in production)
+WHOOSH_JWT_SECRET_FILE=/run/secrets/jwt_secret
+WHOOSH_WEBHOOK_SECRET_FILE=/run/secrets/webhook_secret
+WHOOSH_CORS_ALLOWED_ORIGINS=https://app.company.com,https://admin.company.com
+
+# Gitea
+WHOOSH_GITEA_URL=https://git.company.com
+WHOOSH_GITEA_TOKEN_FILE=/run/secrets/gitea_token
+WHOOSH_GITEA_EAGER_FILTER=true
+
+# External services
+WHOOSH_N8N_BASE_URL=https://workflows.company.com
+WHOOSH_BACKBEAT_URL=http://backbeat:3001
+
+# Observability
+WHOOSH_OTEL_ENABLED=true
+WHOOSH_OTEL_ENDPOINT=http://jaeger:14268/api/traces
+WHOOSH_LOG_LEVEL=info
+
+# Feature flags (conservative defaults)
+WHOOSH_FEATURE_LLM_CLASSIFICATION=false
+WHOOSH_FEATURE_LLM_SKILL_ANALYSIS=false
+
+# Docker
+WHOOSH_DOCKER_ENABLED=true
+```
+
+### Development Environment (.env.development)
+
+```bash
+# Development configuration
+# More permissive settings for local development
+
+# Database
+WHOOSH_DATABASE_URL=postgres://whoosh:password@localhost:5432/whoosh?sslmode=disable
+
+# Security (relaxed for development)
+WHOOSH_JWT_SECRET=dev-secret-change-in-production
+WHOOSH_WEBHOOK_SECRET=dev-webhook-secret
+WHOOSH_CORS_ALLOWED_ORIGINS=http://localhost:3000,http://localhost:8080
+
+# Gitea
+WHOOSH_GITEA_URL=http://localhost:3000
+WHOOSH_GITEA_TOKEN=your-dev-token
+WHOOSH_GITEA_DEBUG_URLS=true
+
+# Logging (verbose for debugging)
+WHOOSH_LOG_LEVEL=debug
+WHOOSH_LOG_CALLER=true
+
+# Feature flags (enable experimental features)
+WHOOSH_FEATURE_LLM_CLASSIFICATION=true
+WHOOSH_FEATURE_ENABLE_ANALYSIS_LOGGING=true
+
+# Docker (disabled for local development)
+WHOOSH_DOCKER_ENABLED=false
+```
+
+## 🔧 Configuration Validation
+
+WHOOSH validates configuration on startup and provides detailed error messages for invalid settings:
+
+### Required Variables
+- `WHOOSH_DATABASE_URL` or individual DB components
+- `WHOOSH_GITEA_URL`
+- `WHOOSH_GITEA_TOKEN` or `WHOOSH_GITEA_TOKEN_FILE`
+
+### Common Validation Errors
+
+```bash
+# Invalid database URL
+ERROR: Invalid database URL format
+
+# Missing secrets
+ERROR: JWT secret not found. Set WHOOSH_JWT_SECRET or WHOOSH_JWT_SECRET_FILE
+
+# Invalid CORS configuration
+ERROR: CORS wildcard (*) not allowed in production. Set specific origins.
+
+# Docker connection failed
+WARNING: Docker not available. Agent deployment disabled.
+```
+
+## 🚀 Best Practices
+
+### Production Deployment
+
+1. **Use Docker secrets** for all sensitive data
+2. **Set specific CORS origins** (never use wildcards)
+3. **Enable rate limiting** and input validation
+4. **Configure appropriate timeouts** for your network
+5. **Enable observability** (tracing, metrics, logs)
+6. **Use conservative feature flags** until tested
+
+### Security Hardening
+
+1. **Rotate secrets regularly** using automated processes
+2. **Use TLS everywhere** in production
+3. **Monitor security logs** for suspicious activity  
+4. **Keep dependency versions updated**
+5. **Review access logs** regularly
+
+### Performance Optimization
+
+1. **Tune database connection pools** based on load
+2. **Configure appropriate cache settings**
+3. **Use CDN for static assets** if applicable
+4. **Monitor resource usage** and scale accordingly
+5. **Enable compression** for large responses
+
+### Troubleshooting
+
+1. **Enable debug logging** temporarily for issues
+2. **Check health endpoints** for component status
+3. **Monitor trace data** for request flow issues
+4. **Validate configuration** before deployment
+5. **Test in staging environment** first
+
+---
+
+## 📚 Related Documentation
+
+- **[Security Audit](../SECURITY_AUDIT_REPORT.md)** - Security implementation details
+- **[API Specification](API_SPECIFICATION.md)** - Complete API reference  
+- **[Database Schema](DATABASE_SCHEMA.md)** - Database structure
+- **[Deployment Guide](DEPLOYMENT.md)** - Production deployment procedures
+
+For additional support, refer to the main [WHOOSH README](../README.md) or create an issue in the repository.