From 827e332e16a3d81bb41e43d4b023c9c5d9f3011f Mon Sep 17 00:00:00 2001 From: Claude Code Date: Sat, 20 Sep 2025 13:21:47 +1000 Subject: [PATCH] Refresh README and add roadmap --- README.md | 452 ++++-------------------------------------------------- 1 file changed, 29 insertions(+), 423 deletions(-) diff --git a/README.md b/README.md index 22e8dbb..528385a 100644 --- a/README.md +++ b/README.md @@ -1,443 +1,49 @@ -# WHOOSH - Autonomous AI Development Teams +# WHOOSH โ€“ Council & Team Orchestration (Beta) -WHOOSH is the orchestration platform for autonomous AI development teams in the CHORUS ecosystem. It transforms project kickoffs through intelligent **Council Formation**, where specialized AI agents collaborate democratically to create comprehensive project foundations and deliver high-quality solutions through consensus-driven processes. +WHOOSH assembles kickoff councils from Design Brief issues and is evolving toward autonomous team orchestration across the CHORUS stack. Council formation/deployment works today, but persistence, telemetry, and self-organising teams are still under construction. -## ๐ŸŽฏ Production Status +## Current Capabilities -**Current Phase**: **PRODUCTION READY** โœ… +- โœ… Gitea Design Brief detection + council composition (`internal/monitor`, `internal/composer`). +- โœ… Docker Swarm agent deployment with role-specific env vars (`internal/orchestrator`). +- โœ… JWT authentication, rate limiting, OpenTelemetry hooks. +- ๐Ÿšง API persistence: REST handlers still return placeholder data while Postgres wiring is finished (`internal/server/server.go`). +- ๐Ÿšง Analysis ingestion: composer relies on heuristic classification; LLM/analysis ingestion is logged but unimplemented (`internal/composer/service.go`). +- ๐Ÿšง Deployment telemetry: results arenโ€™t persisted yet; monitoring includes TODOs for task details (`internal/monitor/monitor.go`). +- ๐Ÿšง Autonomous teams: joining/role balancing planned but not live. -**Council Formation Engine - Fully Implemented:** -- โœ… **Enterprise Security**: JWT authentication, CORS, input validation, rate limiting -- โœ… **Intelligent Council Composition**: Role-based agent deployment from human-roles.yaml -- โœ… **Full API Coverage**: Complete council lifecycle management with artifacts -- โœ… **Production Observability**: OpenTelemetry tracing, comprehensive health monitoring -- โœ… **Configurable Architecture**: All endpoints environment-variable driven -- โœ… **Performance Optimized**: Database indexes, efficient P2P discovery -- ๐Ÿ”„ **Next Phase**: Advanced team evolution for ongoing development tasks +The full plan and sequencing live in: +- `docs/progress/WHOOSH-roadmap.md` +- `docs/DEVELOPMENT_PLAN.md` -## ๐Ÿ—๏ธ Architecture - -### Core Components - -- **Go Backend**: HTTP server with chi framework, structured logging with zerolog, OpenTelemetry tracing -- **PostgreSQL Database**: Council, agent, task, and artifact management with optimized indexes -- **Gitea Integration**: Webhook processing and API client for Design Brief detection -- **Docker Swarm**: Production deployment with CHORUS agent orchestration -- **P2P Network**: Production-ready service discovery and inter-agent communication -- **Security Stack**: JWT auth, CORS, input validation, rate limiting, security headers - -### Council Formation Workflow - -1. **Detection**: Gitea webhook receives "Design Brief" issue with `chorus-entrypoint` label -2. **Analysis**: WHOOSH analyzes project requirements and constraints -3. **Composition**: Intelligent council formation using role definitions from human-roles.yaml -4. **Deployment**: CHORUS agents deployed via Docker Swarm with role-specific configuration -5. **Collaboration**: Agents communicate via P2P network using HMMM protocol -6. **Artifacts**: Council produces kickoff deliverables (manifests, DRs, scaffold plans) -7. **Handoff**: Council artifacts inform subsequent development team formation - -## ๐Ÿš€ Quick Start - -### Prerequisites - -- Docker Swarm cluster -- PostgreSQL database -- Gitea instance -- CHORUS stack deployed - -### Production Deployment +## Quick Start ```bash -# Clone repository git clone https://gitea.chorus.services/tony/WHOOSH.git cd WHOOSH - -# Setup environment configuration cp .env.example .env -# Edit .env with your production settings (see Configuration section) - -# Setup Docker Swarm secrets -echo "your-jwt-secret" | docker secret create whoosh_jwt_secret - -echo "your-gitea-token" | docker secret create whoosh_gitea_token - -echo "your-webhook-secret" | docker secret create whoosh_webhook_secret - - -# Deploy to CHORUS stack -docker stack deploy -c docker-compose.swarm.yml CHORUS -``` - -### Local Development - -```bash -# Install dependencies -go mod download - -# Setup local database -docker-compose up -d postgres - -# Run migrations -migrate -path migrations -database "postgres://whoosh:password@localhost:5432/whoosh?sslmode=disable" up - -# Start development server +# Update DB, JWT, Gitea tokens +make migrate go run ./cmd/whoosh ``` -## โš™๏ธ Configuration +By default the API runs on `:8080` and expects Postgres + Docker Swarm in the environment. Until persistence lands, project/council endpoints return mock payloads to keep the UI working. -Copy and customize the environment configuration: +## Roadmap Snapshot -```bash -cp .env.example .env -# Edit .env with your specific settings -``` +1. **Data path hardening** โ€“ replace mock handlers with real Postgres reads/writes. +2. **Telemetry** โ€“ Persist deployment outcomes, emit KACHING events, build dashboards. +3. **Autonomous loop** โ€“ Drive team formation/joining from composer outputs, tighten HMMM collaboration. +4. **UX & governance** โ€“ Admin dashboards, compliance hooks, Decision Records. -**Core Configuration Options:** +Refer to the roadmap for sprint-by-sprint targets and exit criteria. -```bash -# Database -WHOOSH_DATABASE_URL=postgres://user:pass@localhost/whoosh +## Working With Councils -# Gitea Integration -WHOOSH_GITEA_URL=https://your-gitea-instance.com -WHOOSH_GITEA_TOKEN_FILE=/path/to/gitea-token -WHOOSH_GITEA_EAGER_FILTER=true # API-level vs code-level filtering -WHOOSH_GITEA_FULL_RESCAN=false # Complete vs incremental scanning +- Monitor issues via the API (`GET /api/v1/councils`). +- Inspect generated artifacts (`GET /api/v1/councils/{id}/artifacts`). +- Use Swarm to watch agent containers spin up/down during council execution. -# External Integrations -WHOOSH_N8N_BASE_URL=https://n8n.your-domain.com # Configurable webhook URL +## Contributing -# Security & CORS -WHOOSH_CORS_ALLOWED_ORIGINS=https://app.your-domain.com,https://admin.your-domain.com -WHOOSH_JWT_SECRET_FILE=/path/to/jwt-secret -WHOOSH_WEBHOOK_SECRET_FILE=/path/to/webhook-secret - -# Feature Flags -WHOOSH_FEATURE_LLM_CLASSIFICATION=false # AI vs heuristic classification -WHOOSH_FEATURE_LLM_SKILL_ANALYSIS=false # AI skill analysis - -# Docker Swarm -WHOOSH_DOCKER_ENABLED=true - -# Observability -WHOOSH_BACKBEAT_URL=http://backbeat:3001 -WHOOSH_OTEL_ENABLED=true # OpenTelemetry tracing -``` - -> **๐Ÿ“– Complete Configuration Guide**: See [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md) for all available options. - -### Setup Steps - -1. **Deploy WHOOSH to CHORUS Stack**: - ```bash - docker stack deploy -c docker-compose.swarm.yml CHORUS - ``` - -2. **Configure Repository Monitoring**: - ```bash - curl -X POST http://localhost:8080/api/repositories \ - -H "Content-Type: application/json" \ - -H "Authorization: Bearer ${JWT_TOKEN}" \ - -d '{"full_name": "username/repository", "gitea_id": 123}' - ``` - -3. **Verify System Health**: - ```bash - curl http://localhost:8080/admin/health/details - ``` - -### API Testing - -Check system health: -```bash -curl http://localhost:8080/health -``` - -Get detailed health information: -```bash -curl http://localhost:8080/admin/health/details -``` - -View council status: -```bash -curl -H "Authorization: Bearer ${JWT_TOKEN}" \ - http://localhost:8080/api/v1/councils -``` - -Get specific council details: -```bash -curl -H "Authorization: Bearer ${JWT_TOKEN}" \ - http://localhost:8080/api/v1/councils/{council_id} -``` - -Manage council artifacts: -```bash -# Create artifact -curl -X POST -H "Authorization: Bearer ${JWT_TOKEN}" \ - -H "Content-Type: application/json" \ - -d '{"artifact_type": "kickoff_manifest", "content": "..."}' \ - http://localhost:8080/api/v1/councils/{council_id}/artifacts - -# List artifacts -curl -H "Authorization: Bearer ${JWT_TOKEN}" \ - http://localhost:8080/api/v1/councils/{council_id}/artifacts -``` - -## ๐Ÿ“Š Monitoring & Observability - -### Health Endpoints - -- `/health` - Basic service health -- `/admin/health/details` - Comprehensive component status (DB, Gitea, BackBeat, Docker) -- `/metrics` - Prometheus metrics -- `/debug/vars` - Runtime statistics - -### Distributed Tracing - -WHOOSH includes comprehensive OpenTelemetry integration: -- **Monitor Cycles**: Repository scanning and issue processing spans -- **Council Formation**: Role assignment and composition tracking spans -- **Agent Deployment**: Docker Swarm service creation spans -- **Webhook Processing**: Request/response correlation spans -- **Correlation IDs**: `goal.id` and `pulse.id` attributes for end-to-end tracing - -### Structured Logging - -Enhanced logging with correlation: -- **Request IDs**: HTTP request tracing -- **Council IDs**: Council lifecycle tracking -- **Agent IDs**: Individual agent events -- **Span IDs**: OpenTelemetry trace correlation -- **Performance Metrics**: Operation timing and resource usage - -### BackBeat Integration - -Performance monitoring via BackBeat SDK: -```bash -# View performance metrics -curl http://localhost:8080/api/backbeat/status -``` - -### Monitoring Stack Integration - -**Recommended Tools:** -- **Jaeger/Tempo**: Distributed tracing visualization -- **Prometheus**: Metrics collection and alerting -- **Grafana**: Dashboards and visualization -- **Loki**: Log aggregation and correlation - -## ๐Ÿ” Security - -### Enterprise-Grade Security - -WHOOSH implements comprehensive production security: - -**โœ… Authentication & Authorization:** -- **JWT Authentication**: Role-based access control for web and API access -- **Service Tokens**: Secure API automation with scoped permissions -- **Admin Endpoint Protection**: All administrative operations require authentication - -**โœ… Input Validation & Rate Limiting:** -- **Request Validation**: Comprehensive input sanitization with regex patterns -- **Size Limits**: Configurable request body size limits (1MB default, 10MB webhooks) -- **Rate Limiting**: Per-IP rate limiting (100 requests/minute, configurable) - -**โœ… Security Headers:** -- **CORS**: Configurable domain restrictions (no wildcards in production) -- **CSP**: Content Security Policy headers -- **Security Headers**: X-Frame-Options, X-Content-Type-Options, X-XSS-Protection - -**โœ… Webhook Security:** -- **HMAC Validation**: Enhanced Gitea webhook signature verification -- **Content-Type Validation**: Strict payload type checking -- **Attack Logging**: Comprehensive security event logging - -### Secret Management - -**Production-Ready Secret Handling:** -- **File-based secrets**: Recommended for production (Docker secrets, Kubernetes secrets) -- **Environment variables**: Development and testing -- **Secret validation**: Startup validation ensures all required secrets are present -- **No hardcoded credentials**: All sensitive data externalized - -**Security Audit:** -- **OWASP Top 10**: All major vulnerabilities addressed -- **Security Scan Clean**: Comprehensive security audit completed -- **Compliance Ready**: SOC 2, ISO 27001 preparation - -> **๐Ÿ”’ Security Details**: See [`SECURITY_AUDIT_REPORT.md`](SECURITY_AUDIT_REPORT.md) for complete security analysis. - -## ๐Ÿ“‹ API Reference - -### Core Endpoints - -**Repository Management:** -| Endpoint | Method | Auth | Description | -|----------|--------|------|-------------| -| `/api/repositories` | GET, POST | JWT | Repository monitoring management | -| `/api/repositories/{id}/sync` | POST | JWT | Manual repository synchronization | - -**Council Management:** -| Endpoint | Method | Auth | Description | -|----------|--------|------|-------------| -| `/api/councils` | GET | JWT | List all councils | -| `/api/councils/{id}` | GET | JWT | Get council details with agents | -| `/api/councils/{id}/artifacts` | GET, POST | JWT | Manage council artifacts | - -**Task & Project Management:** -| Endpoint | Method | Auth | Description | -|----------|--------|------|-------------| -| `/api/tasks` | GET, POST | JWT | Task lifecycle management | -| `/api/projects/analyze` | POST | JWT | Project analysis via N8N integration | - -**System & Health:** -| Endpoint | Method | Auth | Description | -|----------|--------|------|-------------| -| `/health` | GET | None | Basic health check | -| `/admin/health/details` | GET | JWT | Comprehensive system status | -| `/metrics` | GET | None | Prometheus metrics | - -**Webhooks:** -| Endpoint | Method | Auth | Description | -|----------|--------|------|-------------| -| `/webhooks/gitea` | POST | HMAC | Gitea webhook handler | - -> **๐Ÿ“š Complete API Documentation**: See [`docs/API_SPECIFICATION.md`](docs/API_SPECIFICATION.md) for detailed API reference. - -## ๐Ÿ—„๏ธ Database Schema - -### Core Tables - -**Council Management:** -- **councils**: Project kickoff council tracking with metadata -- **council_agents**: Individual agent deployment and status -- **council_artifacts**: Deliverable production (manifests, DRs, plans) -- **council_decisions**: Decision-making process and consensus tracking - -**System Tables:** -- **repositories**: Monitored Gitea repositories -- **tasks**: Task lifecycle management -- **agents**: Agent registry and capabilities -- **issues**: Gitea issue synchronization - -**Performance Optimization:** -- Optimized indexes on all hot query paths -- Connection pooling and query optimization -- Migration-based schema management - -> **๐Ÿ“Š Schema Documentation**: See [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) for complete schema reference. - -## ๐Ÿ› ๏ธ Development Commands - -```bash -# Build binary -go build ./cmd/whoosh - -# Run tests -go test ./... - -# Format code -go fmt ./... - -# Static analysis -go vet ./... - -# Database migrations -migrate -path migrations -database "postgres://..." up - -# Docker build -docker build -t whoosh:latest . - -# Security scan -go run honnef.co/go/tools/cmd/staticcheck ./... -``` - -## ๐Ÿ—บ๏ธ Development Roadmap - -### Phase 1: Council Formation Engine โœ… **COMPLETED** -- [x] Design Brief detection and parsing (`chorus-entrypoint` labels) -- [x] Intelligent council composition algorithms -- [x] Production-grade agent deployment via Docker Swarm -- [x] P2P communication infrastructure -- [x] **Production Readiness Achieved**: - - [x] Enterprise security (auth, CORS, input validation, rate limiting) - - [x] Full API coverage (council lifecycle, artifacts management) - - [x] OpenTelemetry distributed tracing with correlation IDs - - [x] Comprehensive health monitoring and observability - - [x] Configurable architecture (no hardcoded dependencies) - - [x] Database performance optimization - - [x] Feature flag system (LLM vs heuristic composition) - -### Phase 2: Enhanced Collaboration ๐Ÿ”„ **IN PROGRESS** -- [x] HMMM protocol foundation -- [ ] Advanced consensus mechanisms and voting -- [ ] Rich artifact template system -- [ ] Cross-council coordination workflows -- [ ] SLURP integration for knowledge preservation - -### Phase 3: Autonomous Teams ๐Ÿ“‹ **PLANNED** -- [ ] Dynamic team formation for ongoing development tasks -- [ ] Self-organizing agent behaviors and capability matching -- [ ] Performance-based team composition optimization -- [ ] BZZZ integration for post-kickoff task management -- [ ] Handoff mechanisms from councils to development teams - -### Phase 4: Advanced Intelligence ๐Ÿ”ฎ **FUTURE** -- [ ] Machine learning for optimal team composition -- [ ] Predictive project success modeling -- [ ] Cross-project knowledge transfer and pattern recognition -- [ ] Advanced quality prediction and risk assessment -- [ ] Multi-project portfolio coordination - -## ๐Ÿ”— Integration Points - -### CHORUS Ecosystem -- **CHORUS Agents**: Specialized AI agents with role-based prompts -- **BZZZ System**: Task management for post-kickoff development -- **SLURP Storage**: Artifact preservation and knowledge graph -- **HMMM Protocol**: Structured inter-agent communication -- **UCXL Addressing**: Decentralized resource identification - -### External Systems -- **Gitea**: Repository hosting and webhook integration -- **N8N**: Workflow automation and LLM analysis -- **Docker Swarm**: Container orchestration and deployment -- **PostgreSQL**: Data persistence and transaction management - -## ๐Ÿ“š Documentation - -- **[API Specification](docs/API_SPECIFICATION.md)** - Complete API reference -- **[Configuration Guide](docs/CONFIGURATION.md)** - Environment variables and setup -- **[Database Schema](docs/DATABASE_SCHEMA.md)** - Schema documentation -- **[Development Plan](docs/DEVELOPMENT_PLAN.md)** - Roadmap and architecture -- **[Security Audit](SECURITY_AUDIT_REPORT.md)** - Security analysis and compliance -- **[Deployment Guide](docs/DEPLOYMENT.md)** - Production deployment procedures - -## ๐Ÿค Contributing - -WHOOSH follows the CHORUS ecosystem development patterns: - -1. **Branch Strategy**: Feature branches from `main` -2. **Development**: Implement with comprehensive tests and documentation -3. **Security**: All changes undergo security review -4. **Deployment**: Version tags trigger automated builds -5. **Validation**: Staging deployment before production -6. **Review**: Detailed PR review process - -### Code Standards -- Go formatting with `gofmt` -- Comprehensive test coverage -- Security-first development -- OpenTelemetry instrumentation -- Structured logging with correlation IDs - -## ๐Ÿ“„ License - -This project is part of the CHORUS ecosystem. All rights reserved. - ---- - -**Production Status**: โœ… **READY FOR DEPLOYMENT** -Built with โค๏ธ using collaborative AI agents, enterprise-grade security, and comprehensive observability. - -**Latest Release**: Production-ready council formation engine with full security, API coverage, and monitoring capabilities. - -**WHOOSH** - *Where autonomous AI development teams come together* ๐ŸŽญ \ No newline at end of file +Before landing features, align with roadmap tickets (`WSH-API`, `WSH-ANALYSIS`, `WSH-OBS`, `WSH-AUTO`, `WSH-UX`). Include Decision Records (UCXL addresses) for architectural/security changes so SLURP/BUBBLE can ingest them later.