Refresh README and add roadmap

2025-09-20 13:21:47 +10:00
parent 7c1c80a8b5
commit 827e332e16
1 changed files with 29 additions and 423 deletions
--- a/README.md
+++ b/README.md
@@ -1,443 +1,49 @@
-# WHOOSH - Autonomous AI Development Teams
+# WHOOSH – Council & Team Orchestration (Beta)

-WHOOSH is the orchestration platform for autonomous AI development teams in the CHORUS ecosystem. It transforms project kickoffs through intelligent **Council Formation**, where specialized AI agents collaborate democratically to create comprehensive project foundations and deliver high-quality solutions through consensus-driven processes.
+WHOOSH assembles kickoff councils from Design Brief issues and is evolving toward autonomous team orchestration across the CHORUS stack. Council formation/deployment works today, but persistence, telemetry, and self-organising teams are still under construction.

-## 🎯 Production Status
+## Current Capabilities

-**Current Phase**: **PRODUCTION READY** ✅
+- ✅ Gitea Design Brief detection + council composition (`internal/monitor`, `internal/composer`).
+- ✅ Docker Swarm agent deployment with role-specific env vars (`internal/orchestrator`).
+- ✅ JWT authentication, rate limiting, OpenTelemetry hooks.
+- 🚧 API persistence: REST handlers still return placeholder data while Postgres wiring is finished (`internal/server/server.go`).
+- 🚧 Analysis ingestion: composer relies on heuristic classification; LLM/analysis ingestion is logged but unimplemented (`internal/composer/service.go`).
+- 🚧 Deployment telemetry: results aren’t persisted yet; monitoring includes TODOs for task details (`internal/monitor/monitor.go`).
+- 🚧 Autonomous teams: joining/role balancing planned but not live.

-**Council Formation Engine - Fully Implemented:**
- ✅ **Enterprise Security**: JWT authentication, CORS, input validation, rate limiting
- ✅ **Intelligent Council Composition**: Role-based agent deployment from human-roles.yaml
- ✅ **Full API Coverage**: Complete council lifecycle management with artifacts
- ✅ **Production Observability**: OpenTelemetry tracing, comprehensive health monitoring
- ✅ **Configurable Architecture**: All endpoints environment-variable driven
- ✅ **Performance Optimized**: Database indexes, efficient P2P discovery
- 🔄 **Next Phase**: Advanced team evolution for ongoing development tasks
+The full plan and sequencing live in:
+- `docs/progress/WHOOSH-roadmap.md`
+- `docs/DEVELOPMENT_PLAN.md`

-## 🏗️ Architecture
-
-### Core Components
-
- **Go Backend**: HTTP server with chi framework, structured logging with zerolog, OpenTelemetry tracing
- **PostgreSQL Database**: Council, agent, task, and artifact management with optimized indexes
- **Gitea Integration**: Webhook processing and API client for Design Brief detection
- **Docker Swarm**: Production deployment with CHORUS agent orchestration
- **P2P Network**: Production-ready service discovery and inter-agent communication
- **Security Stack**: JWT auth, CORS, input validation, rate limiting, security headers
-
-### Council Formation Workflow
-
-1. **Detection**: Gitea webhook receives "Design Brief" issue with `chorus-entrypoint` label
-2. **Analysis**: WHOOSH analyzes project requirements and constraints
-3. **Composition**: Intelligent council formation using role definitions from human-roles.yaml
-4. **Deployment**: CHORUS agents deployed via Docker Swarm with role-specific configuration
-5. **Collaboration**: Agents communicate via P2P network using HMMM protocol
-6. **Artifacts**: Council produces kickoff deliverables (manifests, DRs, scaffold plans)
-7. **Handoff**: Council artifacts inform subsequent development team formation
-
-## 🚀 Quick Start
-
-### Prerequisites
-
- Docker Swarm cluster
- PostgreSQL database
- Gitea instance
- CHORUS stack deployed
-
-### Production Deployment
+## Quick Start

 ```bash
-# Clone repository
 git clone https://gitea.chorus.services/tony/WHOOSH.git
 cd WHOOSH
-
-# Setup environment configuration
 cp .env.example .env
-# Edit .env with your production settings (see Configuration section)
-
-# Setup Docker Swarm secrets
-echo "your-jwt-secret" | docker secret create whoosh_jwt_secret -
-echo "your-gitea-token" | docker secret create whoosh_gitea_token -
-echo "your-webhook-secret" | docker secret create whoosh_webhook_secret -
-
-# Deploy to CHORUS stack
-docker stack deploy -c docker-compose.swarm.yml CHORUS
-```
-
-### Local Development
-
-```bash
-# Install dependencies
-go mod download
-
-# Setup local database
-docker-compose up -d postgres
-
-# Run migrations
-migrate -path migrations -database "postgres://whoosh:password@localhost:5432/whoosh?sslmode=disable" up
-
-# Start development server
+# Update DB, JWT, Gitea tokens
+make migrate
 go run ./cmd/whoosh
 ```

-## ⚙️ Configuration
+By default the API runs on `:8080` and expects Postgres + Docker Swarm in the environment. Until persistence lands, project/council endpoints return mock payloads to keep the UI working.

-Copy and customize the environment configuration:
+## Roadmap Snapshot

-```bash
-cp .env.example .env
-# Edit .env with your specific settings
-```
+1. **Data path hardening** – replace mock handlers with real Postgres reads/writes.
+2. **Telemetry** – Persist deployment outcomes, emit KACHING events, build dashboards.
+3. **Autonomous loop** – Drive team formation/joining from composer outputs, tighten HMMM collaboration.
+4. **UX & governance** – Admin dashboards, compliance hooks, Decision Records.

-**Core Configuration Options:**
+Refer to the roadmap for sprint-by-sprint targets and exit criteria.

-```bash
-# Database
-WHOOSH_DATABASE_URL=postgres://user:pass@localhost/whoosh
+## Working With Councils

-# Gitea Integration  
-WHOOSH_GITEA_URL=https://your-gitea-instance.com
-WHOOSH_GITEA_TOKEN_FILE=/path/to/gitea-token
-WHOOSH_GITEA_EAGER_FILTER=true     # API-level vs code-level filtering
-WHOOSH_GITEA_FULL_RESCAN=false     # Complete vs incremental scanning
+- Monitor issues via the API (`GET /api/v1/councils`).
+- Inspect generated artifacts (`GET /api/v1/councils/{id}/artifacts`).
+- Use Swarm to watch agent containers spin up/down during council execution.

-# External Integrations
-WHOOSH_N8N_BASE_URL=https://n8n.your-domain.com  # Configurable webhook URL
+## Contributing

-# Security & CORS
-WHOOSH_CORS_ALLOWED_ORIGINS=https://app.your-domain.com,https://admin.your-domain.com
-WHOOSH_JWT_SECRET_FILE=/path/to/jwt-secret
-WHOOSH_WEBHOOK_SECRET_FILE=/path/to/webhook-secret
-
-# Feature Flags
-WHOOSH_FEATURE_LLM_CLASSIFICATION=false  # AI vs heuristic classification
-WHOOSH_FEATURE_LLM_SKILL_ANALYSIS=false  # AI skill analysis
-
-# Docker Swarm
-WHOOSH_DOCKER_ENABLED=true
-
-# Observability
-WHOOSH_BACKBEAT_URL=http://backbeat:3001
-WHOOSH_OTEL_ENABLED=true              # OpenTelemetry tracing
-```
-
-> **📖 Complete Configuration Guide**: See [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md) for all available options.
-
-### Setup Steps
-
-1. **Deploy WHOOSH to CHORUS Stack**:
-   ```bash
-   docker stack deploy -c docker-compose.swarm.yml CHORUS
-   ```
-
-2. **Configure Repository Monitoring**:
-   ```bash
-   curl -X POST http://localhost:8080/api/repositories \
-     -H "Content-Type: application/json" \
-     -H "Authorization: Bearer ${JWT_TOKEN}" \
-     -d '{"full_name": "username/repository", "gitea_id": 123}'
-   ```
-
-3. **Verify System Health**:
-   ```bash
-   curl http://localhost:8080/admin/health/details
-   ```
-
-### API Testing
-
-Check system health:
-```bash
-curl http://localhost:8080/health
-```
-
-Get detailed health information:
-```bash
-curl http://localhost:8080/admin/health/details
-```
-
-View council status:
-```bash  
-curl -H "Authorization: Bearer ${JWT_TOKEN}" \
-  http://localhost:8080/api/v1/councils
-```
-
-Get specific council details:
-```bash
-curl -H "Authorization: Bearer ${JWT_TOKEN}" \
-  http://localhost:8080/api/v1/councils/{council_id}
-```
-
-Manage council artifacts:
-```bash
-# Create artifact
-curl -X POST -H "Authorization: Bearer ${JWT_TOKEN}" \
-  -H "Content-Type: application/json" \
-  -d '{"artifact_type": "kickoff_manifest", "content": "..."}' \
-  http://localhost:8080/api/v1/councils/{council_id}/artifacts
-
-# List artifacts
-curl -H "Authorization: Bearer ${JWT_TOKEN}" \
-  http://localhost:8080/api/v1/councils/{council_id}/artifacts
-```
-
-## 📊 Monitoring & Observability
-
-### Health Endpoints
-
- `/health` - Basic service health
- `/admin/health/details` - Comprehensive component status (DB, Gitea, BackBeat, Docker)
- `/metrics` - Prometheus metrics
- `/debug/vars` - Runtime statistics
-
-### Distributed Tracing
-
-WHOOSH includes comprehensive OpenTelemetry integration:
- **Monitor Cycles**: Repository scanning and issue processing spans
- **Council Formation**: Role assignment and composition tracking spans
- **Agent Deployment**: Docker Swarm service creation spans
- **Webhook Processing**: Request/response correlation spans
- **Correlation IDs**: `goal.id` and `pulse.id` attributes for end-to-end tracing
-
-### Structured Logging
-
-Enhanced logging with correlation:
- **Request IDs**: HTTP request tracing
- **Council IDs**: Council lifecycle tracking
- **Agent IDs**: Individual agent events
- **Span IDs**: OpenTelemetry trace correlation
- **Performance Metrics**: Operation timing and resource usage
-
-### BackBeat Integration
-
-Performance monitoring via BackBeat SDK:
-```bash
-# View performance metrics
-curl http://localhost:8080/api/backbeat/status
-```
-
-### Monitoring Stack Integration
-
-**Recommended Tools:**
- **Jaeger/Tempo**: Distributed tracing visualization
- **Prometheus**: Metrics collection and alerting
- **Grafana**: Dashboards and visualization
- **Loki**: Log aggregation and correlation
-
-## 🔐 Security
-
-### Enterprise-Grade Security
-
-WHOOSH implements comprehensive production security:
-
-**✅ Authentication & Authorization:**
- **JWT Authentication**: Role-based access control for web and API access
- **Service Tokens**: Secure API automation with scoped permissions
- **Admin Endpoint Protection**: All administrative operations require authentication
-
-**✅ Input Validation & Rate Limiting:**
- **Request Validation**: Comprehensive input sanitization with regex patterns
- **Size Limits**: Configurable request body size limits (1MB default, 10MB webhooks)
- **Rate Limiting**: Per-IP rate limiting (100 requests/minute, configurable)
-
-**✅ Security Headers:**
- **CORS**: Configurable domain restrictions (no wildcards in production)
- **CSP**: Content Security Policy headers
- **Security Headers**: X-Frame-Options, X-Content-Type-Options, X-XSS-Protection
-
-**✅ Webhook Security:**
- **HMAC Validation**: Enhanced Gitea webhook signature verification
- **Content-Type Validation**: Strict payload type checking
- **Attack Logging**: Comprehensive security event logging
-
-### Secret Management
-
-**Production-Ready Secret Handling:**
- **File-based secrets**: Recommended for production (Docker secrets, Kubernetes secrets)
- **Environment variables**: Development and testing
- **Secret validation**: Startup validation ensures all required secrets are present
- **No hardcoded credentials**: All sensitive data externalized
-
-**Security Audit:**
- **OWASP Top 10**: All major vulnerabilities addressed
- **Security Scan Clean**: Comprehensive security audit completed
- **Compliance Ready**: SOC 2, ISO 27001 preparation
-
-> **🔒 Security Details**: See [`SECURITY_AUDIT_REPORT.md`](SECURITY_AUDIT_REPORT.md) for complete security analysis.
-
-## 📋 API Reference
-
-### Core Endpoints
-
-**Repository Management:**
-| Endpoint | Method | Auth | Description |
-|----------|--------|------|-------------|
-| `/api/repositories` | GET, POST | JWT | Repository monitoring management |
-| `/api/repositories/{id}/sync` | POST | JWT | Manual repository synchronization |
-
-**Council Management:**
-| Endpoint | Method | Auth | Description |
-|----------|--------|------|-------------|
-| `/api/councils` | GET | JWT | List all councils |
-| `/api/councils/{id}` | GET | JWT | Get council details with agents |
-| `/api/councils/{id}/artifacts` | GET, POST | JWT | Manage council artifacts |
-
-**Task & Project Management:**
-| Endpoint | Method | Auth | Description |
-|----------|--------|------|-------------|
-| `/api/tasks` | GET, POST | JWT | Task lifecycle management |
-| `/api/projects/analyze` | POST | JWT | Project analysis via N8N integration |
-
-**System & Health:**
-| Endpoint | Method | Auth | Description |
-|----------|--------|------|-------------|
-| `/health` | GET | None | Basic health check |
-| `/admin/health/details` | GET | JWT | Comprehensive system status |
-| `/metrics` | GET | None | Prometheus metrics |
-
-**Webhooks:**
-| Endpoint | Method | Auth | Description |
-|----------|--------|------|-------------|
-| `/webhooks/gitea` | POST | HMAC | Gitea webhook handler |
-
-> **📚 Complete API Documentation**: See [`docs/API_SPECIFICATION.md`](docs/API_SPECIFICATION.md) for detailed API reference.
-
-## 🗄️ Database Schema
-
-### Core Tables
-
-**Council Management:**
- **councils**: Project kickoff council tracking with metadata
- **council_agents**: Individual agent deployment and status
- **council_artifacts**: Deliverable production (manifests, DRs, plans)
- **council_decisions**: Decision-making process and consensus tracking
-
-**System Tables:**
- **repositories**: Monitored Gitea repositories
- **tasks**: Task lifecycle management
- **agents**: Agent registry and capabilities
- **issues**: Gitea issue synchronization
-
-**Performance Optimization:**
- Optimized indexes on all hot query paths
- Connection pooling and query optimization
- Migration-based schema management
-
-> **📊 Schema Documentation**: See [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) for complete schema reference.
-
-## 🛠️ Development Commands
-
-```bash
-# Build binary
-go build ./cmd/whoosh
-
-# Run tests
-go test ./...
-
-# Format code
-go fmt ./...
-
-# Static analysis
-go vet ./...
-
-# Database migrations
-migrate -path migrations -database "postgres://..." up
-
-# Docker build
-docker build -t whoosh:latest .
-
-# Security scan
-go run honnef.co/go/tools/cmd/staticcheck ./...
-```
-
-## 🗺️ Development Roadmap
-
-### Phase 1: Council Formation Engine ✅ **COMPLETED**
- [x] Design Brief detection and parsing (`chorus-entrypoint` labels)
- [x] Intelligent council composition algorithms
- [x] Production-grade agent deployment via Docker Swarm
- [x] P2P communication infrastructure
- [x] **Production Readiness Achieved**:
-  - [x] Enterprise security (auth, CORS, input validation, rate limiting)
-  - [x] Full API coverage (council lifecycle, artifacts management)
-  - [x] OpenTelemetry distributed tracing with correlation IDs
-  - [x] Comprehensive health monitoring and observability
-  - [x] Configurable architecture (no hardcoded dependencies)
-  - [x] Database performance optimization
-  - [x] Feature flag system (LLM vs heuristic composition)
-
-### Phase 2: Enhanced Collaboration 🔄 **IN PROGRESS**
- [x] HMMM protocol foundation
- [ ] Advanced consensus mechanisms and voting
- [ ] Rich artifact template system
- [ ] Cross-council coordination workflows
- [ ] SLURP integration for knowledge preservation
-
-### Phase 3: Autonomous Teams 📋 **PLANNED**
- [ ] Dynamic team formation for ongoing development tasks
- [ ] Self-organizing agent behaviors and capability matching
- [ ] Performance-based team composition optimization
- [ ] BZZZ integration for post-kickoff task management
- [ ] Handoff mechanisms from councils to development teams
-
-### Phase 4: Advanced Intelligence 🔮 **FUTURE**
- [ ] Machine learning for optimal team composition
- [ ] Predictive project success modeling
- [ ] Cross-project knowledge transfer and pattern recognition
- [ ] Advanced quality prediction and risk assessment
- [ ] Multi-project portfolio coordination
-
-## 🔗 Integration Points
-
-### CHORUS Ecosystem
- **CHORUS Agents**: Specialized AI agents with role-based prompts
- **BZZZ System**: Task management for post-kickoff development
- **SLURP Storage**: Artifact preservation and knowledge graph
- **HMMM Protocol**: Structured inter-agent communication
- **UCXL Addressing**: Decentralized resource identification
-
-### External Systems
- **Gitea**: Repository hosting and webhook integration
- **N8N**: Workflow automation and LLM analysis
- **Docker Swarm**: Container orchestration and deployment
- **PostgreSQL**: Data persistence and transaction management
-
-## 📚 Documentation
-
- **[API Specification](docs/API_SPECIFICATION.md)** - Complete API reference
- **[Configuration Guide](docs/CONFIGURATION.md)** - Environment variables and setup
- **[Database Schema](docs/DATABASE_SCHEMA.md)** - Schema documentation
- **[Development Plan](docs/DEVELOPMENT_PLAN.md)** - Roadmap and architecture
- **[Security Audit](SECURITY_AUDIT_REPORT.md)** - Security analysis and compliance
- **[Deployment Guide](docs/DEPLOYMENT.md)** - Production deployment procedures
-
-## 🤝 Contributing
-
-WHOOSH follows the CHORUS ecosystem development patterns:
-
-1. **Branch Strategy**: Feature branches from `main`
-2. **Development**: Implement with comprehensive tests and documentation
-3. **Security**: All changes undergo security review
-4. **Deployment**: Version tags trigger automated builds
-5. **Validation**: Staging deployment before production
-6. **Review**: Detailed PR review process
-
-### Code Standards
- Go formatting with `gofmt`
- Comprehensive test coverage
- Security-first development
- OpenTelemetry instrumentation
- Structured logging with correlation IDs
-
-## 📄 License
-
-This project is part of the CHORUS ecosystem. All rights reserved.
-
---
-
-**Production Status**: ✅ **READY FOR DEPLOYMENT**  
-Built with ❤️ using collaborative AI agents, enterprise-grade security, and comprehensive observability.
-
-**Latest Release**: Production-ready council formation engine with full security, API coverage, and monitoring capabilities.
-
-**WHOOSH** - *Where autonomous AI development teams come together* 🎭
+Before landing features, align with roadmap tickets (`WSH-API`, `WSH-ANALYSIS`, `WSH-OBS`, `WSH-AUTO`, `WSH-UX`). Include Decision Records (UCXL addresses) for architectural/security changes so SLURP/BUBBLE can ingest them later.