tony/WHOOSH
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Update README and add comprehensive CONFIGURATION guide

Browse Source 
## Documentation Updates

### README.md - Production Status Update
- Changed status from "MVP → Production Ready Transition" to "PRODUCTION READY ✅"
- Added comprehensive Council Formation workflow (7-step process)
- Updated architecture components with security stack
- Enhanced API reference with authentication requirements
- Added production deployment instructions
- Comprehensive security section with enterprise-grade features
- OpenTelemetry tracing and observability documentation
- Updated development roadmap with phase completion status

### CONFIGURATION.md - New Comprehensive Guide
- Complete reference for 60+ environment variables
- Categorized sections: Database, Security, External Services, Feature Flags
- Production and development configuration templates
- Security best practices and hardening recommendations
- Validation guide with common errors and troubleshooting
- Performance tuning recommendations

## Key Highlights
- Production-ready status clearly communicated
- All new security features documented
- Complete configuration management guide
- Enterprise deployment procedures
- Comprehensive observability setup

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Claude Code
2025-09-12 22:55:27 +10:00
parent 131868bdca
commit e5555ae277
2 changed files with 816 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files

466
README.md
View File

@@ -1,116 +1,329 @@
# WHOOSH - Autonomous AI Development Teams
WHOOSH is the orchestration platform for autonomous AI development teams in the CHORUS ecosystem. It transforms from a simple project template tool into a sophisticated system that enables AI agents to form optimal teams, collaborate democratically, and deliver high-quality solutions through consensus-driven development processes.
WHOOSH is the orchestration platform for autonomous AI development teams in the CHORUS ecosystem. It transforms project kickoffs through intelligent **Council Formation**, where specialized AI agents collaborate democratically to create comprehensive project foundations and deliver high-quality solutions through consensus-driven processes.
## 🎯 MVP Goals
## 🎯 Production Status
The current MVP focuses on:
**Current Phase**: **PRODUCTION READY** ✅
1. **Single-Agent Execution**: Process `bzzz-task` labeled issues with single-agent teams
2. **GITEA Integration**: Webhook handling for task discovery and PR management
3. **Basic Team Management**: Minimal team state tracking and assignment
4. **SLURP Integration**: Artifact submission and retrieval proxy
5. **Docker Swarm Deployment**: Production-ready containerization
**Council Formation Engine - Fully Implemented:**
- ✅ **Enterprise Security**: JWT authentication, CORS, input validation, rate limiting
- ✅ **Intelligent Council Composition**: Role-based agent deployment from human-roles.yaml
- ✅ **Full API Coverage**: Complete council lifecycle management with artifacts
- ✅ **Production Observability**: OpenTelemetry tracing, comprehensive health monitoring
- ✅ **Configurable Architecture**: All endpoints environment-variable driven
- ✅ **Performance Optimized**: Database indexes, efficient P2P discovery
- 🔄 **Next Phase**: Advanced team evolution for ongoing development tasks
## 🏗️ Architecture
### Core Components
- **Go Backend**: HTTP server with chi/echo framework, structured logging with zerolog
- **PostgreSQL Database**: Team, agent, and task state management with migrations
- **GITEA Integration**: Webhook processing and API client for issue management
- **Docker Swarm**: Production deployment with secrets management
- **Redis**: Optional caching and session management
- **Go Backend**: HTTP server with chi framework, structured logging with zerolog, OpenTelemetry tracing
- **PostgreSQL Database**: Council, agent, task, and artifact management with optimized indexes
- **Gitea Integration**: Webhook processing and API client for Design Brief detection
- **Docker Swarm**: Production deployment with CHORUS agent orchestration
- **P2P Network**: Production-ready service discovery and inter-agent communication
- **Security Stack**: JWT auth, CORS, input validation, rate limiting, security headers
### MVP Workflow
### Council Formation Workflow
1. GITEA webhook receives issue with `bzzz-task` label
2. WHOOSH parses task information and creates team assignment
3. Single-agent executor processes task (stubbed Team Composer)
4. Results submitted via SLURP proxy for artifact preservation
5. PR creation and status updates back to GITEA
1. **Detection**: Gitea webhook receives "Design Brief" issue with `chorus-entrypoint` label
2. **Analysis**: WHOOSH analyzes project requirements and constraints
3. **Composition**: Intelligent council formation using role definitions from human-roles.yaml
4. **Deployment**: CHORUS agents deployed via Docker Swarm with role-specific configuration
5. **Collaboration**: Agents communicate via P2P network using HMMM protocol
6. **Artifacts**: Council produces kickoff deliverables (manifests, DRs, scaffold plans)
7. **Handoff**: Council artifacts inform subsequent development team formation
## 🚀 Quick Start
### Local Development
### Prerequisites
- Docker Swarm cluster
- PostgreSQL database
- Gitea instance
- CHORUS stack deployed
### Production Deployment
```bash
# Clone repository
git clone https://gitea.chorus.services/tony/WHOOSH.git
cd WHOOSH
# Copy environment configuration
# Setup environment configuration
cp .env.example .env
# Edit .env with your configuration
# Edit .env with your production settings (see Configuration section)
# Start with Docker Compose
docker-compose up -d
# Setup Docker Swarm secrets
echo "your-jwt-secret" | docker secret create whoosh_jwt_secret -
echo "your-gitea-token" | docker secret create whoosh_gitea_token -
echo "your-webhook-secret" | docker secret create whoosh_webhook_secret -
# Or run locally
# Deploy to CHORUS stack
docker stack deploy -c docker-compose.swarm.yml CHORUS
```
### Local Development
```bash
# Install dependencies
go mod download
# Setup local database
docker-compose up -d postgres
# Run migrations
migrate -path migrations -database "postgres://whoosh:password@localhost:5432/whoosh?sslmode=disable" up
# Start development server
go run ./cmd/whoosh
```
### Production Deployment
## ⚙️ Configuration
Copy and customize the environment configuration:
```bash
# Setup Docker Swarm secrets
./scripts/setup-secrets.sh
# Deploy to swarm
./scripts/deploy-swarm.sh v0.1.0-mvp
cp .env.example .env
# Edit .env with your specific settings
```
## 📋 API Endpoints
**Core Configuration Options:**
### Health & Status
- `GET /health` - Service health check
- `GET /health/ready` - Readiness check with database connection
```bash
# Database
WHOOSH_DATABASE_URL=postgres://user:pass@localhost/whoosh
### Teams (MVP Minimal)
- `GET /api/v1/teams` - List teams
- `POST /api/v1/teams` - Create team (stub)
- `GET /api/v1/teams/{teamID}` - Get team details (stub)
- `PUT /api/v1/teams/{teamID}/status` - Update team status (stub)
# Gitea Integration
WHOOSH_GITEA_URL=https://your-gitea-instance.com
WHOOSH_GITEA_TOKEN_FILE=/path/to/gitea-token
WHOOSH_GITEA_EAGER_FILTER=true # API-level vs code-level filtering
WHOOSH_GITEA_FULL_RESCAN=false # Complete vs incremental scanning
### Task Management
- `POST /api/v1/tasks/ingest` - Task ingestion (stub)
- `GET /api/v1/tasks/{taskID}` - Get task details (stub)
# External Integrations
WHOOSH_N8N_BASE_URL=https://n8n.your-domain.com # Configurable webhook URL
### SLURP Integration
- `POST /api/v1/slurp/submit` - Submit artifacts (stub)
- `GET /api/v1/slurp/artifacts/{ucxlAddr}` - Retrieve artifacts (stub)
# Security & CORS
WHOOSH_CORS_ALLOWED_ORIGINS=https://app.your-domain.com,https://admin.your-domain.com
WHOOSH_JWT_SECRET_FILE=/path/to/jwt-secret
WHOOSH_WEBHOOK_SECRET_FILE=/path/to/webhook-secret
### CHORUS Integration
- `GET /api/v1/projects/{projectID}/tasks` - List project tasks
- `GET /api/v1/projects/{projectID}/tasks/available` - List available tasks
- `GET /api/v1/projects/{projectID}/repository` - Get project repository info
- `GET /api/v1/projects/{projectID}/tasks/{taskNumber}` - Get specific task
- `POST /api/v1/projects/{projectID}/tasks/{taskNumber}/claim` - Claim task for agent
- `PUT /api/v1/projects/{projectID}/tasks/{taskNumber}/status` - Update task status
- `POST /api/v1/projects/{projectID}/tasks/{taskNumber}/complete` - Complete task
- `POST /api/v1/agents/register` - Register CHORUS agent
- `PUT /api/v1/agents/{agentID}/status` - Update agent status
# Feature Flags
WHOOSH_FEATURE_LLM_CLASSIFICATION=false # AI vs heuristic classification
WHOOSH_FEATURE_LLM_SKILL_ANALYSIS=false # AI skill analysis
### Webhooks
- `POST /webhooks/gitea` - GITEA webhook endpoint (implemented)
# Docker Swarm
WHOOSH_DOCKER_ENABLED=true
# Observability
WHOOSH_BACKBEAT_URL=http://backbeat:3001
WHOOSH_OTEL_ENABLED=true # OpenTelemetry tracing
```
> **📖 Complete Configuration Guide**: See [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md) for all available options.
### Setup Steps
1. **Deploy WHOOSH to CHORUS Stack**:
```bash
docker stack deploy -c docker-compose.swarm.yml CHORUS
```
2. **Configure Repository Monitoring**:
```bash
curl -X POST http://localhost:8080/api/repositories \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${JWT_TOKEN}" \
-d '{"full_name": "username/repository", "gitea_id": 123}'
```
3. **Verify System Health**:
```bash
curl http://localhost:8080/admin/health/details
```
### API Testing
Check system health:
```bash
curl http://localhost:8080/health
```
Get detailed health information:
```bash
curl http://localhost:8080/admin/health/details
```
View council status:
```bash
curl -H "Authorization: Bearer ${JWT_TOKEN}" \
http://localhost:8080/api/v1/councils
```
Get specific council details:
```bash
curl -H "Authorization: Bearer ${JWT_TOKEN}" \
http://localhost:8080/api/v1/councils/{council_id}
```
Manage council artifacts:
```bash
# Create artifact
curl -X POST -H "Authorization: Bearer ${JWT_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"artifact_type": "kickoff_manifest", "content": "..."}' \
http://localhost:8080/api/v1/councils/{council_id}/artifacts
# List artifacts
curl -H "Authorization: Bearer ${JWT_TOKEN}" \
http://localhost:8080/api/v1/councils/{council_id}/artifacts
```
## 📊 Monitoring & Observability
### Health Endpoints
- `/health` - Basic service health
- `/admin/health/details` - Comprehensive component status (DB, Gitea, BackBeat, Docker)
- `/metrics` - Prometheus metrics
- `/debug/vars` - Runtime statistics
### Distributed Tracing
WHOOSH includes comprehensive OpenTelemetry integration:
- **Monitor Cycles**: Repository scanning and issue processing spans
- **Council Formation**: Role assignment and composition tracking spans
- **Agent Deployment**: Docker Swarm service creation spans
- **Webhook Processing**: Request/response correlation spans
- **Correlation IDs**: `goal.id` and `pulse.id` attributes for end-to-end tracing
### Structured Logging
Enhanced logging with correlation:
- **Request IDs**: HTTP request tracing
- **Council IDs**: Council lifecycle tracking
- **Agent IDs**: Individual agent events
- **Span IDs**: OpenTelemetry trace correlation
- **Performance Metrics**: Operation timing and resource usage
### BackBeat Integration
Performance monitoring via BackBeat SDK:
```bash
# View performance metrics
curl http://localhost:8080/api/backbeat/status
```
### Monitoring Stack Integration
**Recommended Tools:**
- **Jaeger/Tempo**: Distributed tracing visualization
- **Prometheus**: Metrics collection and alerting
- **Grafana**: Dashboards and visualization
- **Loki**: Log aggregation and correlation
## 🔐 Security
### Enterprise-Grade Security
WHOOSH implements comprehensive production security:
**✅ Authentication & Authorization:**
- **JWT Authentication**: Role-based access control for web and API access
- **Service Tokens**: Secure API automation with scoped permissions
- **Admin Endpoint Protection**: All administrative operations require authentication
**✅ Input Validation & Rate Limiting:**
- **Request Validation**: Comprehensive input sanitization with regex patterns
- **Size Limits**: Configurable request body size limits (1MB default, 10MB webhooks)
- **Rate Limiting**: Per-IP rate limiting (100 requests/minute, configurable)
**✅ Security Headers:**
- **CORS**: Configurable domain restrictions (no wildcards in production)
- **CSP**: Content Security Policy headers
- **Security Headers**: X-Frame-Options, X-Content-Type-Options, X-XSS-Protection
**✅ Webhook Security:**
- **HMAC Validation**: Enhanced Gitea webhook signature verification
- **Content-Type Validation**: Strict payload type checking
- **Attack Logging**: Comprehensive security event logging
### Secret Management
**Production-Ready Secret Handling:**
- **File-based secrets**: Recommended for production (Docker secrets, Kubernetes secrets)
- **Environment variables**: Development and testing
- **Secret validation**: Startup validation ensures all required secrets are present
- **No hardcoded credentials**: All sensitive data externalized
**Security Audit:**
- **OWASP Top 10**: All major vulnerabilities addressed
- **Security Scan Clean**: Comprehensive security audit completed
- **Compliance Ready**: SOC 2, ISO 27001 preparation
> **🔒 Security Details**: See [`SECURITY_AUDIT_REPORT.md`](SECURITY_AUDIT_REPORT.md) for complete security analysis.
## 📋 API Reference
### Core Endpoints
**Repository Management:**
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/api/repositories` | GET, POST | JWT | Repository monitoring management |
| `/api/repositories/{id}/sync` | POST | JWT | Manual repository synchronization |
**Council Management:**
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/api/councils` | GET | JWT | List all councils |
| `/api/councils/{id}` | GET | JWT | Get council details with agents |
| `/api/councils/{id}/artifacts` | GET, POST | JWT | Manage council artifacts |
**Task & Project Management:**
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/api/tasks` | GET, POST | JWT | Task lifecycle management |
| `/api/projects/analyze` | POST | JWT | Project analysis via N8N integration |
**System & Health:**
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/health` | GET | None | Basic health check |
| `/admin/health/details` | GET | JWT | Comprehensive system status |
| `/metrics` | GET | None | Prometheus metrics |
**Webhooks:**
| Endpoint | Method | Auth | Description |
|----------|--------|------|-------------|
| `/webhooks/gitea` | POST | HMAC | Gitea webhook handler |
> **📚 Complete API Documentation**: See [`docs/API_SPECIFICATION.md`](docs/API_SPECIFICATION.md) for detailed API reference.
## 🗄️ Database Schema
### Core Tables (MVP)
### Core Tables
- **teams**: Team management and status tracking
- **team_roles**: Available roles (executor, coordinator, reviewer)
- **team_assignments**: Agent-to-team assignments
- **agents**: Minimal agent registry
- **slurp_submissions**: Artifact tracking
**Council Management:**
- **councils**: Project kickoff council tracking with metadata
- **council_agents**: Individual agent deployment and status
- **council_artifacts**: Deliverable production (manifests, DRs, plans)
- **council_decisions**: Decision-making process and consensus tracking
## 🔐 Security Features
**System Tables:**
- **repositories**: Monitored Gitea repositories
- **tasks**: Task lifecycle management
- **agents**: Agent registry and capabilities
- **issues**: Gitea issue synchronization
- **Docker Swarm Secrets**: Sensitive data management
- **SHHH Integration**: Data redaction and encryption
- **JWT Authentication**: Service and user token validation
- **Webhook Signature Validation**: GITEA webhook authenticity
- **Rate Limiting**: API endpoint protection
**Performance Optimization:**
- Optimized indexes on all hot query paths
- Connection pooling and query optimization
- Migration-based schema management
> **📊 Schema Documentation**: See [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) for complete schema reference.
## 🛠️ Development Commands
@@ -129,62 +342,92 @@ go vet ./...
# Database migrations
migrate -path migrations -database "postgres://..." up
# Docker build
docker build -t whoosh:latest .
# Security scan
go run honnef.co/go/tools/cmd/staticcheck ./...
```
## 📊 Monitoring
## 🗺️ Development Roadmap
### Docker Swarm Services
### Phase 1: Council Formation Engine ✅ **COMPLETED**
- [x] Design Brief detection and parsing (`chorus-entrypoint` labels)
- [x] Intelligent council composition algorithms
- [x] Production-grade agent deployment via Docker Swarm
- [x] P2P communication infrastructure
- [x] **Production Readiness Achieved**:
- [x] Enterprise security (auth, CORS, input validation, rate limiting)
- [x] Full API coverage (council lifecycle, artifacts management)
- [x] OpenTelemetry distributed tracing with correlation IDs
- [x] Comprehensive health monitoring and observability
- [x] Configurable architecture (no hardcoded dependencies)
- [x] Database performance optimization
- [x] Feature flag system (LLM vs heuristic composition)
```bash
# Service status
docker service ls --filter label=com.docker.stack.namespace=whoosh
### Phase 2: Enhanced Collaboration 🔄 **IN PROGRESS**
- [x] HMMM protocol foundation
- [ ] Advanced consensus mechanisms and voting
- [ ] Rich artifact template system
- [ ] Cross-council coordination workflows
- [ ] SLURP integration for knowledge preservation
# Service logs
docker service logs -f whoosh_whoosh
### Phase 3: Autonomous Teams 📋 **PLANNED**
- [ ] Dynamic team formation for ongoing development tasks
- [ ] Self-organizing agent behaviors and capability matching
- [ ] Performance-based team composition optimization
- [ ] BZZZ integration for post-kickoff task management
- [ ] Handoff mechanisms from councils to development teams
# Scale services
docker service scale whoosh_whoosh=3
```
### Phase 4: Advanced Intelligence 🔮 **FUTURE**
- [ ] Machine learning for optimal team composition
- [ ] Predictive project success modeling
- [ ] Cross-project knowledge transfer and pattern recognition
- [ ] Advanced quality prediction and risk assessment
- [ ] Multi-project portfolio coordination
### Health Endpoints
## 🔗 Integration Points
- Health: `https://whoosh.chorus.services/health`
- Ready: `https://whoosh.chorus.services/health/ready`
## 🔄 Future Roadmap
### Post-MVP Features
1. **Team Composer**: LLM-powered task analysis and team formation
2. **P2P Communication**: UCXL addressing and HMMM integration
3. **Agent Self-Organization**: Automatic team application and consensus
4. **Advanced Analytics**: Performance metrics and team effectiveness
5. **Multi-Repository Support**: Cross-project team coordination
### Integration Points
- **CHORUS Agents**: P2P task coordination and execution
- **BZZZ System**: Distributed task management integration
- **SHHH Encryption**: Secure data handling and transmission
### CHORUS Ecosystem
- **CHORUS Agents**: Specialized AI agents with role-based prompts
- **BZZZ System**: Task management for post-kickoff development
- **SLURP Storage**: Artifact preservation and knowledge graph
- **HMMM Protocol**: Structured inter-agent communication
- **UCXL Addressing**: Decentralized resource identification
- **SLURP Storage**: Comprehensive artifact preservation
### External Systems
- **Gitea**: Repository hosting and webhook integration
- **N8N**: Workflow automation and LLM analysis
- **Docker Swarm**: Container orchestration and deployment
- **PostgreSQL**: Data persistence and transaction management
## 📚 Documentation
- [Development Plan](docs/DEVELOPMENT_PLAN.md) - Comprehensive transformation roadmap
- [Database Schema](docs/DATABASE_SCHEMA.md) - Complete schema documentation
- [API Specification](docs/API_SPECIFICATION.md) - Full API reference
- [Team Composer Spec](docs/TEAM_COMPOSER_SPEC.md) - LLM integration details
- **[API Specification](docs/API_SPECIFICATION.md)** - Complete API reference
- **[Configuration Guide](docs/CONFIGURATION.md)** - Environment variables and setup
- **[Database Schema](docs/DATABASE_SCHEMA.md)** - Schema documentation
- **[Development Plan](docs/DEVELOPMENT_PLAN.md)** - Roadmap and architecture
- **[Security Audit](SECURITY_AUDIT_REPORT.md)** - Security analysis and compliance
- **[Deployment Guide](docs/DEPLOYMENT.md)** - Production deployment procedures
## 🤝 Contributing
WHOOSH follows the CHORUS ecosystem development patterns:
1. Branch from `main` for features
2. Implement with comprehensive tests
3. Update version tags for container builds
4. Deploy to staging for validation
5. Create PR with detailed description
1. **Branch Strategy**: Feature branches from `main`
2. **Development**: Implement with comprehensive tests and documentation
3. **Security**: All changes undergo security review
4. **Deployment**: Version tags trigger automated builds
5. **Validation**: Staging deployment before production
6. **Review**: Detailed PR review process
### Code Standards
- Go formatting with `gofmt`
- Comprehensive test coverage
- Security-first development
- OpenTelemetry instrumentation
- Structured logging with correlation IDs
## 📄 License
@@ -192,4 +435,9 @@ This project is part of the CHORUS ecosystem. All rights reserved.
---
**Production Status**: ✅ **READY FOR DEPLOYMENT**
Built with ❤️ using collaborative AI agents, enterprise-grade security, and comprehensive observability.
**Latest Release**: Production-ready council formation engine with full security, API coverage, and monitoring capabilities.
**WHOOSH** - *Where autonomous AI development teams come together* 🎭