WHOOSH/README.md

# WHOOSH - Autonomous AI Development Teams

WHOOSH is the orchestration platform for autonomous AI development teams in the CHORUS ecosystem. It transforms project kickoffs through intelligent **Council Formation**, where specialized AI agents collaborate democratically to create comprehensive project foundations and deliver high-quality solutions through consensus-driven processes.

## 🎯 Production Status

**Current Phase**: **PRODUCTION READY** ✅

**Council Formation Engine - Fully Implemented:**
- ✅ **Enterprise Security**: JWT authentication, CORS, input validation, rate limiting
- ✅ **Intelligent Council Composition**: Role-based agent deployment from human-roles.yaml
- ✅ **Full API Coverage**: Complete council lifecycle management with artifacts
- ✅ **Production Observability**: OpenTelemetry tracing, comprehensive health monitoring
- ✅ **Configurable Architecture**: All endpoints environment-variable driven
- ✅ **Performance Optimized**: Database indexes, efficient P2P discovery
- 🔄 **Next Phase**: Advanced team evolution for ongoing development tasks

## 🏗️ Architecture

### Core Components

- **Go Backend**: HTTP server with chi framework, structured logging with zerolog, OpenTelemetry tracing
- **PostgreSQL Database**: Council, agent, task, and artifact management with optimized indexes
- **Gitea Integration**: Webhook processing and API client for Design Brief detection
- **Docker Swarm**: Production deployment with CHORUS agent orchestration
- **P2P Network**: Production-ready service discovery and inter-agent communication
- **Security Stack**: JWT auth, CORS, input validation, rate limiting, security headers

### Council Formation Workflow

1. **Detection**: Gitea webhook receives "Design Brief" issue with `chorus-entrypoint` label
2. **Analysis**: WHOOSH analyzes project requirements and constraints
3. **Composition**: Intelligent council formation using role definitions from human-roles.yaml
4. **Deployment**: CHORUS agents deployed via Docker Swarm with role-specific configuration
5. **Collaboration**: Agents communicate via P2P network using HMMM protocol
6. **Artifacts**: Council produces kickoff deliverables (manifests, DRs, scaffold plans)
7. **Handoff**: Council artifacts inform subsequent development team formation

## 🚀 Quick Start

### Prerequisites

- Docker Swarm cluster
- PostgreSQL database
- Gitea instance
- CHORUS stack deployed

### Production Deployment

```bash
# Clone repository
git clone https://gitea.chorus.services/tony/WHOOSH.git
cd WHOOSH

# Setup environment configuration
cp .env.example .env
# Edit .env with your production settings (see Configuration section)

# Setup Docker Swarm secrets
echo "your-jwt-secret" | docker secret create whoosh_jwt_secret -
echo "your-gitea-token" | docker secret create whoosh_gitea_token -
echo "your-webhook-secret" | docker secret create whoosh_webhook_secret -

# Deploy to CHORUS stack
docker stack deploy -c docker-compose.swarm.yml CHORUS
```

### Local Development

```bash
# Install dependencies
go mod download

# Setup local database
docker-compose up -d postgres

# Run migrations
migrate -path migrations -database "postgres://whoosh:password@localhost:5432/whoosh?sslmode=disable" up

# Start development server
go run ./cmd/whoosh
```

## ⚙️ Configuration

Copy and customize the environment configuration:

```bash
cp .env.example .env
# Edit .env with your specific settings
```

**Core Configuration Options:**

```bash
# Database
WHOOSH_DATABASE_URL=postgres://user:pass@localhost/whoosh

# Gitea Integration  
WHOOSH_GITEA_URL=https://your-gitea-instance.com
WHOOSH_GITEA_TOKEN_FILE=/path/to/gitea-token
WHOOSH_GITEA_EAGER_FILTER=true     # API-level vs code-level filtering
WHOOSH_GITEA_FULL_RESCAN=false     # Complete vs incremental scanning

# External Integrations
WHOOSH_N8N_BASE_URL=https://n8n.your-domain.com  # Configurable webhook URL

# Security & CORS
WHOOSH_CORS_ALLOWED_ORIGINS=https://app.your-domain.com,https://admin.your-domain.com
WHOOSH_JWT_SECRET_FILE=/path/to/jwt-secret
WHOOSH_WEBHOOK_SECRET_FILE=/path/to/webhook-secret

# Feature Flags
WHOOSH_FEATURE_LLM_CLASSIFICATION=false  # AI vs heuristic classification
WHOOSH_FEATURE_LLM_SKILL_ANALYSIS=false  # AI skill analysis

# Docker Swarm
WHOOSH_DOCKER_ENABLED=true

# Observability
WHOOSH_BACKBEAT_URL=http://backbeat:3001
WHOOSH_OTEL_ENABLED=true              # OpenTelemetry tracing
```

> **📖 Complete Configuration Guide**: See [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md) for all available options.

### Setup Steps

1. **Deploy WHOOSH to CHORUS Stack**:
   ```bash
   docker stack deploy -c docker-compose.swarm.yml CHORUS
   ```

2. **Configure Repository Monitoring**:
   ```bash
   curl -X POST http://localhost:8080/api/repositories \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer ${JWT_TOKEN}" \
     -d '{"full_name": "username/repository", "gitea_id": 123}'
   ```

3. **Verify System Health**:
   ```bash
   curl http://localhost:8080/admin/health/details
   ```

### API Testing

Check system health:
```bash
curl http://localhost:8080/health
```

Get detailed health information:
```bash
curl http://localhost:8080/admin/health/details
```

View council status:
```bash  
curl -H "Authorization: Bearer ${JWT_TOKEN}" \
  http://localhost:8080/api/v1/councils
```

Get specific council details:
```bash
curl -H "Authorization: Bearer ${JWT_TOKEN}" \
  http://localhost:8080/api/v1/councils/{council_id}
```

Manage council artifacts:
```bash
# Create artifact
curl -X POST -H "Authorization: Bearer ${JWT_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"artifact_type": "kickoff_manifest", "content": "..."}' \
  http://localhost:8080/api/v1/councils/{council_id}/artifacts

# List artifacts
curl -H "Authorization: Bearer ${JWT_TOKEN}" \
  http://localhost:8080/api/v1/councils/{council_id}/artifacts
```

## 📊 Monitoring & Observability

### Health Endpoints

- `/health` - Basic service health
- `/admin/health/details` - Comprehensive component status (DB, Gitea, BackBeat, Docker)
- `/metrics` - Prometheus metrics
- `/debug/vars` - Runtime statistics

### Distributed Tracing

WHOOSH includes comprehensive OpenTelemetry integration:
- **Monitor Cycles**: Repository scanning and issue processing spans
- **Council Formation**: Role assignment and composition tracking spans
- **Agent Deployment**: Docker Swarm service creation spans
- **Webhook Processing**: Request/response correlation spans
- **Correlation IDs**: `goal.id` and `pulse.id` attributes for end-to-end tracing

### Structured Logging

Enhanced logging with correlation:
- **Request IDs**: HTTP request tracing
- **Council IDs**: Council lifecycle tracking
- **Agent IDs**: Individual agent events
- **Span IDs**: OpenTelemetry trace correlation
- **Performance Metrics**: Operation timing and resource usage

### BackBeat Integration

Performance monitoring via BackBeat SDK:
```bash
# View performance metrics
curl http://localhost:8080/api/backbeat/status
```

### Monitoring Stack Integration

**Recommended Tools:**
- **Jaeger/Tempo**: Distributed tracing visualization
- **Prometheus**: Metrics collection and alerting
- **Grafana**: Dashboards and visualization
- **Loki**: Log aggregation and correlation

## 🔐 Security

### Enterprise-Grade Security

WHOOSH implements comprehensive production security:

**✅ Authentication & Authorization:**
- **JWT Authentication**: Role-based access control for web and API access
- **Service Tokens**: Secure API automation with scoped permissions
- **Admin Endpoint Protection**: All administrative operations require authentication

**✅ Input Validation & Rate Limiting:**
- **Request Validation**: Comprehensive input sanitization with regex patterns
- **Size Limits**: Configurable request body size limits (1MB default, 10MB webhooks)
- **Rate Limiting**: Per-IP rate limiting (100 requests/minute, configurable)

**✅ Security Headers:**
- **CORS**: Configurable domain restrictions (no wildcards in production)
- **CSP**: Content Security Policy headers
- **Security Headers**: X-Frame-Options, X-Content-Type-Options, X-XSS-Protection

**✅ Webhook Security:**
- **HMAC Validation**: Enhanced Gitea webhook signature verification
- **Content-Type Validation**: Strict payload type checking
- **Attack Logging**: Comprehensive security event logging

### Secret Management

**Production-Ready Secret Handling:**
- **File-based secrets**: Recommended for production (Docker secrets, Kubernetes secrets)
- **Environment variables**: Development and testing
- **Secret validation**: Startup validation ensures all required secrets are present
- **No hardcoded credentials**: All sensitive data externalized

**Security Audit:**
- **OWASP Top 10**: All major vulnerabilities addressed
- **Security Scan Clean**: Comprehensive security audit completed
- **Compliance Ready**: SOC 2, ISO 27001 preparation

> **🔒 Security Details**: See [`SECURITY_AUDIT_REPORT.md`](SECURITY_AUDIT_REPORT.md) for complete security analysis.

## 📋 API Reference

### Core Endpoints

**Repository Management:**
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/api/repositories` | GET, POST | JWT | Repository monitoring management |
| `/api/repositories/{id}/sync` | POST | JWT | Manual repository synchronization |

**Council Management:**
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/api/councils` | GET | JWT | List all councils |
| `/api/councils/{id}` | GET | JWT | Get council details with agents |
| `/api/councils/{id}/artifacts` | GET, POST | JWT | Manage council artifacts |

**Task & Project Management:**
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/api/tasks` | GET, POST | JWT | Task lifecycle management |
| `/api/projects/analyze` | POST | JWT | Project analysis via N8N integration |

**System & Health:**
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/health` | GET | None | Basic health check |
| `/admin/health/details` | GET | JWT | Comprehensive system status |
| `/metrics` | GET | None | Prometheus metrics |

**Webhooks:**
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/webhooks/gitea` | POST | HMAC | Gitea webhook handler |

> **📚 Complete API Documentation**: See [`docs/API_SPECIFICATION.md`](docs/API_SPECIFICATION.md) for detailed API reference.

## 🗄️ Database Schema

### Core Tables

**Council Management:**
- **councils**: Project kickoff council tracking with metadata
- **council_agents**: Individual agent deployment and status
- **council_artifacts**: Deliverable production (manifests, DRs, plans)
- **council_decisions**: Decision-making process and consensus tracking

**System Tables:**
- **repositories**: Monitored Gitea repositories
- **tasks**: Task lifecycle management
- **agents**: Agent registry and capabilities
- **issues**: Gitea issue synchronization

**Performance Optimization:**
- Optimized indexes on all hot query paths
- Connection pooling and query optimization
- Migration-based schema management

> **📊 Schema Documentation**: See [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) for complete schema reference.

## 🛠️ Development Commands

```bash
# Build binary
go build ./cmd/whoosh

# Run tests
go test ./...

# Format code
go fmt ./...

# Static analysis
go vet ./...

# Database migrations
migrate -path migrations -database "postgres://..." up

# Docker build
docker build -t whoosh:latest .

# Security scan
go run honnef.co/go/tools/cmd/staticcheck ./...
```

## 🗺️ Development Roadmap

### Phase 1: Council Formation Engine ✅ **COMPLETED**
- [x] Design Brief detection and parsing (`chorus-entrypoint` labels)
- [x] Intelligent council composition algorithms
- [x] Production-grade agent deployment via Docker Swarm
- [x] P2P communication infrastructure
- [x] **Production Readiness Achieved**:
  - [x] Enterprise security (auth, CORS, input validation, rate limiting)
  - [x] Full API coverage (council lifecycle, artifacts management)
  - [x] OpenTelemetry distributed tracing with correlation IDs
  - [x] Comprehensive health monitoring and observability
  - [x] Configurable architecture (no hardcoded dependencies)
  - [x] Database performance optimization
  - [x] Feature flag system (LLM vs heuristic composition)

### Phase 2: Enhanced Collaboration 🔄 **IN PROGRESS**
- [x] HMMM protocol foundation
- [ ] Advanced consensus mechanisms and voting
- [ ] Rich artifact template system
- [ ] Cross-council coordination workflows
- [ ] SLURP integration for knowledge preservation

### Phase 3: Autonomous Teams 📋 **PLANNED**
- [ ] Dynamic team formation for ongoing development tasks
- [ ] Self-organizing agent behaviors and capability matching
- [ ] Performance-based team composition optimization
- [ ] BZZZ integration for post-kickoff task management
- [ ] Handoff mechanisms from councils to development teams

### Phase 4: Advanced Intelligence 🔮 **FUTURE**
- [ ] Machine learning for optimal team composition
- [ ] Predictive project success modeling
- [ ] Cross-project knowledge transfer and pattern recognition
- [ ] Advanced quality prediction and risk assessment
- [ ] Multi-project portfolio coordination

## 🔗 Integration Points

### CHORUS Ecosystem
- **CHORUS Agents**: Specialized AI agents with role-based prompts
- **BZZZ System**: Task management for post-kickoff development
- **SLURP Storage**: Artifact preservation and knowledge graph
- **HMMM Protocol**: Structured inter-agent communication
- **UCXL Addressing**: Decentralized resource identification

### External Systems
- **Gitea**: Repository hosting and webhook integration
- **N8N**: Workflow automation and LLM analysis
- **Docker Swarm**: Container orchestration and deployment
- **PostgreSQL**: Data persistence and transaction management

## 📚 Documentation

- **[API Specification](docs/API_SPECIFICATION.md)** - Complete API reference
- **[Configuration Guide](docs/CONFIGURATION.md)** - Environment variables and setup
- **[Database Schema](docs/DATABASE_SCHEMA.md)** - Schema documentation
- **[Development Plan](docs/DEVELOPMENT_PLAN.md)** - Roadmap and architecture
- **[Security Audit](SECURITY_AUDIT_REPORT.md)** - Security analysis and compliance
- **[Deployment Guide](docs/DEPLOYMENT.md)** - Production deployment procedures

## 🤝 Contributing

WHOOSH follows the CHORUS ecosystem development patterns:

1. **Branch Strategy**: Feature branches from `main`
2. **Development**: Implement with comprehensive tests and documentation
3. **Security**: All changes undergo security review
4. **Deployment**: Version tags trigger automated builds
5. **Validation**: Staging deployment before production
6. **Review**: Detailed PR review process

### Code Standards
- Go formatting with `gofmt`
- Comprehensive test coverage
- Security-first development
- OpenTelemetry instrumentation
- Structured logging with correlation IDs

## 📄 License

This project is part of the CHORUS ecosystem. All rights reserved.

---

**Production Status**: ✅ **READY FOR DEPLOYMENT**  
Built with ❤️ using collaborative AI agents, enterprise-grade security, and comprehensive observability.

**Latest Release**: Production-ready council formation engine with full security, API coverage, and monitoring capabilities.

**WHOOSH** - *Where autonomous AI development teams come together* 🎭