WHOOSH/README.md

WHOOSH - Autonomous AI Development Teams

WHOOSH is the orchestration platform for autonomous AI development teams in the CHORUS ecosystem. It transforms project kickoffs through intelligent Council Formation, where specialized AI agents collaborate democratically to create comprehensive project foundations and deliver high-quality solutions through consensus-driven processes.

🎯 Production Status

Current Phase: PRODUCTION READY ✅

Council Formation Engine - Fully Implemented:

• ✅ Enterprise Security: JWT authentication, CORS, input validation, rate limiting
• ✅ Intelligent Council Composition: Role-based agent deployment from human-roles.yaml
• ✅ Full API Coverage: Complete council lifecycle management with artifacts
• ✅ Production Observability: OpenTelemetry tracing, comprehensive health monitoring
• ✅ Configurable Architecture: All endpoints environment-variable driven
• ✅ Performance Optimized: Database indexes, efficient P2P discovery
• 🔄 Next Phase: Advanced team evolution for ongoing development tasks

🏗️ Architecture

Core Components

• Go Backend: HTTP server with chi framework, structured logging with zerolog, OpenTelemetry tracing
• PostgreSQL Database: Council, agent, task, and artifact management with optimized indexes
• Gitea Integration: Webhook processing and API client for Design Brief detection
• Docker Swarm: Production deployment with CHORUS agent orchestration
• P2P Network: Production-ready service discovery and inter-agent communication
• Security Stack: JWT auth, CORS, input validation, rate limiting, security headers

Council Formation Workflow

1. Detection: Gitea webhook receives "Design Brief" issue with chorus-entrypoint label
2. Analysis: WHOOSH analyzes project requirements and constraints
3. Composition: Intelligent council formation using role definitions from human-roles.yaml
4. Deployment: CHORUS agents deployed via Docker Swarm with role-specific configuration
5. Collaboration: Agents communicate via P2P network using HMMM protocol
6. Artifacts: Council produces kickoff deliverables (manifests, DRs, scaffold plans)
7. Handoff: Council artifacts inform subsequent development team formation

🚀 Quick Start

Prerequisites

• Docker Swarm cluster
• PostgreSQL database
• Gitea instance
• CHORUS stack deployed

Production Deployment

# Clone repository
git clone https://gitea.chorus.services/tony/WHOOSH.git
cd WHOOSH

# Setup environment configuration
cp .env.example .env
# Edit .env with your production settings (see Configuration section)

# Setup Docker Swarm secrets
echo "your-jwt-secret" | docker secret create whoosh_jwt_secret -
echo "your-gitea-token" | docker secret create whoosh_gitea_token -
echo "your-webhook-secret" | docker secret create whoosh_webhook_secret -

# Deploy to CHORUS stack
docker stack deploy -c docker-compose.swarm.yml CHORUS

Local Development

# Install dependencies
go mod download

# Setup local database
docker-compose up -d postgres

# Run migrations
migrate -path migrations -database "postgres://whoosh:password@localhost:5432/whoosh?sslmode=disable" up

# Start development server
go run ./cmd/whoosh

⚙️ Configuration

Copy and customize the environment configuration:

cp .env.example .env
# Edit .env with your specific settings

Core Configuration Options:

# Database
WHOOSH_DATABASE_URL=postgres://user:pass@localhost/whoosh

# Gitea Integration  
WHOOSH_GITEA_URL=https://your-gitea-instance.com
WHOOSH_GITEA_TOKEN_FILE=/path/to/gitea-token
WHOOSH_GITEA_EAGER_FILTER=true     # API-level vs code-level filtering
WHOOSH_GITEA_FULL_RESCAN=false     # Complete vs incremental scanning

# External Integrations
WHOOSH_N8N_BASE_URL=https://n8n.your-domain.com  # Configurable webhook URL

# Security & CORS
WHOOSH_CORS_ALLOWED_ORIGINS=https://app.your-domain.com,https://admin.your-domain.com
WHOOSH_JWT_SECRET_FILE=/path/to/jwt-secret
WHOOSH_WEBHOOK_SECRET_FILE=/path/to/webhook-secret

# Feature Flags
WHOOSH_FEATURE_LLM_CLASSIFICATION=false  # AI vs heuristic classification
WHOOSH_FEATURE_LLM_SKILL_ANALYSIS=false  # AI skill analysis

# Docker Swarm
WHOOSH_DOCKER_ENABLED=true

# Observability
WHOOSH_BACKBEAT_URL=http://backbeat:3001
WHOOSH_OTEL_ENABLED=true              # OpenTelemetry tracing

📖 Complete Configuration Guide: See docs/CONFIGURATION.md for all available options.

Setup Steps

1. Deploy WHOOSH to CHORUS Stack:

   docker stack deploy -c docker-compose.swarm.yml CHORUS
   

2. Configure Repository Monitoring:

   curl -X POST http://localhost:8080/api/repositories \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer ${JWT_TOKEN}" \
     -d '{"full_name": "username/repository", "gitea_id": 123}'
   

3. Verify System Health:

   curl http://localhost:8080/admin/health/details
   

API Testing

Check system health:

curl http://localhost:8080/health

Get detailed health information:

curl http://localhost:8080/admin/health/details

View council status:

curl -H "Authorization: Bearer ${JWT_TOKEN}" \
  http://localhost:8080/api/v1/councils

Get specific council details:

curl -H "Authorization: Bearer ${JWT_TOKEN}" \
  http://localhost:8080/api/v1/councils/{council_id}

Manage council artifacts:

# Create artifact
curl -X POST -H "Authorization: Bearer ${JWT_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"artifact_type": "kickoff_manifest", "content": "..."}' \
  http://localhost:8080/api/v1/councils/{council_id}/artifacts

# List artifacts
curl -H "Authorization: Bearer ${JWT_TOKEN}" \
  http://localhost:8080/api/v1/councils/{council_id}/artifacts

📊 Monitoring & Observability

Health Endpoints

• /health - Basic service health
• /admin/health/details - Comprehensive component status (DB, Gitea, BackBeat, Docker)
• /metrics - Prometheus metrics
• /debug/vars - Runtime statistics

Distributed Tracing

WHOOSH includes comprehensive OpenTelemetry integration:

• Monitor Cycles: Repository scanning and issue processing spans
• Council Formation: Role assignment and composition tracking spans
• Agent Deployment: Docker Swarm service creation spans
• Webhook Processing: Request/response correlation spans
• Correlation IDs: goal.id and pulse.id attributes for end-to-end tracing

Structured Logging

Enhanced logging with correlation:

• Request IDs: HTTP request tracing
• Council IDs: Council lifecycle tracking
• Agent IDs: Individual agent events
• Span IDs: OpenTelemetry trace correlation
• Performance Metrics: Operation timing and resource usage

BackBeat Integration

Performance monitoring via BackBeat SDK:

# View performance metrics
curl http://localhost:8080/api/backbeat/status

Monitoring Stack Integration

Recommended Tools:

• Jaeger/Tempo: Distributed tracing visualization
• Prometheus: Metrics collection and alerting
• Grafana: Dashboards and visualization
• Loki: Log aggregation and correlation

🔐 Security

Enterprise-Grade Security

WHOOSH implements comprehensive production security:

✅ Authentication & Authorization:

• JWT Authentication: Role-based access control for web and API access
• Service Tokens: Secure API automation with scoped permissions
• Admin Endpoint Protection: All administrative operations require authentication

✅ Input Validation & Rate Limiting:

• Request Validation: Comprehensive input sanitization with regex patterns
• Size Limits: Configurable request body size limits (1MB default, 10MB webhooks)
• Rate Limiting: Per-IP rate limiting (100 requests/minute, configurable)

✅ Security Headers:

• CORS: Configurable domain restrictions (no wildcards in production)
• CSP: Content Security Policy headers
• Security Headers: X-Frame-Options, X-Content-Type-Options, X-XSS-Protection

✅ Webhook Security:

• HMAC Validation: Enhanced Gitea webhook signature verification
• Content-Type Validation: Strict payload type checking
• Attack Logging: Comprehensive security event logging

Secret Management

Production-Ready Secret Handling:

• File-based secrets: Recommended for production (Docker secrets, Kubernetes secrets)
• Environment variables: Development and testing
• Secret validation: Startup validation ensures all required secrets are present
• No hardcoded credentials: All sensitive data externalized

Security Audit:

• OWASP Top 10: All major vulnerabilities addressed
• Security Scan Clean: Comprehensive security audit completed
• Compliance Ready: SOC 2, ISO 27001 preparation

🔒 Security Details: See SECURITY_AUDIT_REPORT.md for complete security analysis.

📋 API Reference

Core Endpoints

Repository Management:

Endpoint	Method	Auth	Description
/api/repositories	GET, POST	JWT	Repository monitoring management
/api/repositories/{id}/sync	POST	JWT	Manual repository synchronization

Council Management:

Endpoint	Method	Auth	Description
/api/councils	GET	JWT	List all councils
/api/councils/{id}	GET	JWT	Get council details with agents
/api/councils/{id}/artifacts	GET, POST	JWT	Manage council artifacts

Task & Project Management:

Endpoint	Method	Auth	Description
/api/tasks	GET, POST	JWT	Task lifecycle management
/api/projects/analyze	POST	JWT	Project analysis via N8N integration

System & Health:

Endpoint	Method	Auth	Description
/health	GET	None	Basic health check
/admin/health/details	GET	JWT	Comprehensive system status
/metrics	GET	None	Prometheus metrics

Webhooks:

Endpoint	Method	Auth	Description
/webhooks/gitea	POST	HMAC	Gitea webhook handler

📚 Complete API Documentation: See docs/API_SPECIFICATION.md for detailed API reference.

🗄️ Database Schema

Core Tables

Council Management:

• councils: Project kickoff council tracking with metadata
• council_agents: Individual agent deployment and status
• council_artifacts: Deliverable production (manifests, DRs, plans)
• council_decisions: Decision-making process and consensus tracking

System Tables:

• repositories: Monitored Gitea repositories
• tasks: Task lifecycle management
• agents: Agent registry and capabilities
• issues: Gitea issue synchronization

Performance Optimization:

• Optimized indexes on all hot query paths
• Connection pooling and query optimization
• Migration-based schema management

📊 Schema Documentation: See docs/DATABASE_SCHEMA.md for complete schema reference.

🛠️ Development Commands

# Build binary
go build ./cmd/whoosh

# Run tests
go test ./...

# Format code
go fmt ./...

# Static analysis
go vet ./...

# Database migrations
migrate -path migrations -database "postgres://..." up

# Docker build
docker build -t whoosh:latest .

# Security scan
go run honnef.co/go/tools/cmd/staticcheck ./...

🗺️ Development Roadmap

Phase 1: Council Formation Engine ✅ COMPLETED

• Design Brief detection and parsing (chorus-entrypoint labels)
• Intelligent council composition algorithms
• Production-grade agent deployment via Docker Swarm
• P2P communication infrastructure
• Production Readiness Achieved:
  • Enterprise security (auth, CORS, input validation, rate limiting)
  • Full API coverage (council lifecycle, artifacts management)
  • OpenTelemetry distributed tracing with correlation IDs
  • Comprehensive health monitoring and observability
  • Configurable architecture (no hardcoded dependencies)
  • Database performance optimization
  • Feature flag system (LLM vs heuristic composition)

Phase 2: Enhanced Collaboration 🔄 IN PROGRESS

• HMMM protocol foundation
• Advanced consensus mechanisms and voting
• Rich artifact template system
• Cross-council coordination workflows
• SLURP integration for knowledge preservation

Phase 3: Autonomous Teams 📋 PLANNED

• Dynamic team formation for ongoing development tasks
• Self-organizing agent behaviors and capability matching
• Performance-based team composition optimization
• BZZZ integration for post-kickoff task management
• Handoff mechanisms from councils to development teams

Phase 4: Advanced Intelligence 🔮 FUTURE

• Machine learning for optimal team composition
• Predictive project success modeling
• Cross-project knowledge transfer and pattern recognition
• Advanced quality prediction and risk assessment
• Multi-project portfolio coordination

🔗 Integration Points

CHORUS Ecosystem

• CHORUS Agents: Specialized AI agents with role-based prompts
• BZZZ System: Task management for post-kickoff development
• SLURP Storage: Artifact preservation and knowledge graph
• HMMM Protocol: Structured inter-agent communication
• UCXL Addressing: Decentralized resource identification

External Systems

• Gitea: Repository hosting and webhook integration
• N8N: Workflow automation and LLM analysis
• Docker Swarm: Container orchestration and deployment
• PostgreSQL: Data persistence and transaction management

📚 Documentation

• API Specification - Complete API reference
• Configuration Guide - Environment variables and setup
• Database Schema - Schema documentation
• Development Plan - Roadmap and architecture
• Security Audit - Security analysis and compliance
• Deployment Guide - Production deployment procedures

🤝 Contributing

WHOOSH follows the CHORUS ecosystem development patterns:

1. Branch Strategy: Feature branches from main
2. Development: Implement with comprehensive tests and documentation
3. Security: All changes undergo security review
4. Deployment: Version tags trigger automated builds
5. Validation: Staging deployment before production
6. Review: Detailed PR review process

Code Standards

• Go formatting with gofmt
• Comprehensive test coverage
• Security-first development
• OpenTelemetry instrumentation
• Structured logging with correlation IDs

📄 License

This project is part of the CHORUS ecosystem. All rights reserved.

---

Production Status: ✅ READY FOR DEPLOYMENT
Built with ❤️ using collaborative AI agents, enterprise-grade security, and comprehensive observability.

Latest Release: Production-ready council formation engine with full security, API coverage, and monitoring capabilities.

WHOOSH - Where autonomous AI development teams come together 🎭