Complete CHORUS Services integration with unified orchestration
- Docker Compose orchestration for all services - Unified management script (chorus.sh) with full lifecycle controls - Monitoring stack (Prometheus on 9092, Grafana on 3002) - Database initialization and networking configuration - Comprehensive documentation and usage guides - Port conflict resolution (avoid Cockpit on 9090) Components integrated: - WHOOSH: Orchestration platform (ports 3001, 8087) - BZZZ: P2P coordination system (port 8080) - SLURP: Context management system (port 8088) - COOEE: RL feedback system (port 8089) - Infrastructure: PostgreSQL, Redis, monitoring stack Ready for deployment and development.
This commit is contained in:
tony
331
README.md
Normal file
331
README.md
Normal file
@@ -0,0 +1,331 @@
|
||||
# 🎵 CHORUS Services - Distributed AI Orchestration Platform
|
||||
|
||||
**CHORUS Services** eliminates context loss, reduces hallucinations, and enables scalable multi-agent collaboration through intelligent context management and distributed reasoning.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
```bash
|
||||
# Initialize submodules (first time only)
|
||||
./chorus.sh init
|
||||
|
||||
# Start all services
|
||||
./chorus.sh start
|
||||
|
||||
# Check service health
|
||||
./chorus.sh health
|
||||
```
|
||||
|
||||
**Access Points:**
|
||||
- 🌐 **WHOOSH Dashboard**: http://localhost:3001
|
||||
- 📡 **API Documentation**: http://localhost:8087/docs
|
||||
- 📊 **Grafana Monitoring**: http://localhost:3002 (admin/chorusadmin)
|
||||
- 🔍 **Prometheus Metrics**: http://localhost:9092
|
||||
|
||||
## 🏗️ Architecture Overview
|
||||
|
||||
CHORUS Services integrates five core components into a unified platform:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ CHORUS Services Platform │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ 🎛️ WHOOSH Orchestrator │ 📊 Monitoring & Analytics │
|
||||
│ - Workflow Management │ - Prometheus Metrics │
|
||||
│ - Agent Coordination │ - Grafana Dashboards │
|
||||
│ - Task Distribution │ - Real-time Health Monitoring │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ 🐝 BZZZ P2P Network │ 🧠 SLURP Context Management │
|
||||
│ - Agent Mesh Networking │ - Hierarchical Context Storage │
|
||||
│ - Peer Discovery │ - Semantic Search & Indexing │
|
||||
│ - Distributed Coordination │ - Multi-language SDK Support │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ 🎯 COOEE Feedback System (RL Context SLURP Integration) │
|
||||
│ - Performance-based Learning │ - Context Relevance Tuning │
|
||||
│ - Agent Feedback Collection │ - Role-based Access Control │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 🧩 Core Components
|
||||
|
||||
### 🎛️ WHOOSH - Orchestration Engine
|
||||
- **Enterprise workflow management** for AI agents
|
||||
- **Visual workflow editor** with React Flow
|
||||
- **Real-time performance monitoring** and metrics
|
||||
- **Multi-agent task distribution** and coordination
|
||||
|
||||
### 🐝 BZZZ - P2P Agent Coordination
|
||||
- **Mesh networking** with libp2p for resilient communication
|
||||
- **Automatic peer discovery** via mDNS
|
||||
- **Distributed task coordination** without single points of failure
|
||||
- **Go-based** high-performance networking layer
|
||||
|
||||
### 🧠 SLURP - Context Management System
|
||||
- **Hierarchical context storage** with path-based organization
|
||||
- **Hybrid search** combining semantic similarity + BM25
|
||||
- **Multi-language SDKs** (Python, JavaScript, Go, Rust, Java, C#)
|
||||
- **Enterprise-grade APIs** with authentication and WebSocket streaming
|
||||
|
||||
### 🎯 COOEE - Feedback & Learning (RL Context SLURP)
|
||||
- **Reinforcement learning** for context relevance tuning
|
||||
- **Agent feedback collection** with upvote/downvote systems
|
||||
- **Role-based context filtering** and access control
|
||||
- **Continuous improvement** through real-world performance data
|
||||
|
||||
### 📊 Monitoring & Analytics
|
||||
- **Prometheus metrics collection** across all services
|
||||
- **Grafana dashboards** for visualization and alerting
|
||||
- **Health checks** and performance monitoring
|
||||
- **Audit trails** with complete traceability
|
||||
|
||||
## 🛠️ Management Commands
|
||||
|
||||
The `./chorus.sh` script provides unified management:
|
||||
|
||||
```bash
|
||||
# Service Management
|
||||
./chorus.sh start # Start all services
|
||||
./chorus.sh stop # Stop all services
|
||||
./chorus.sh restart # Restart all services
|
||||
./chorus.sh status # Show service status
|
||||
|
||||
# Development & Maintenance
|
||||
./chorus.sh build # Build all docker images
|
||||
./chorus.sh update # Update submodules to latest
|
||||
./chorus.sh logs [service] # View logs
|
||||
./chorus.sh health # Check service health
|
||||
./chorus.sh clean # Clean up resources
|
||||
|
||||
# First-time Setup
|
||||
./chorus.sh init # Initialize git submodules
|
||||
```
|
||||
|
||||
## 🌐 Service Endpoints
|
||||
|
||||
| Service | Port | Purpose | Health Check |
|
||||
|---------|------|---------|--------------|
|
||||
| **WHOOSH Dashboard** | 3001 | Web UI for orchestration | http://localhost:3001 |
|
||||
| **WHOOSH API** | 8087 | REST API + WebSocket | http://localhost:8087/health |
|
||||
| **BZZZ Coordinator** | 8080 | P2P coordination API | http://localhost:8080/health |
|
||||
| **SLURP API** | 8088 | Context management API | http://localhost:8088/health |
|
||||
| **COOEE RL Tuner** | 8089 | Feedback & learning API | http://localhost:8089/health |
|
||||
| **Grafana** | 3002 | Monitoring dashboards | http://localhost:3002 |
|
||||
| **Prometheus** | 9092 | Metrics collection | http://localhost:9092 |
|
||||
| **PostgreSQL** | 5433 | Database | - |
|
||||
| **Redis** | 6380 | Cache & message queue | - |
|
||||
|
||||
## 📁 Project Structure
|
||||
|
||||
```
|
||||
chorus.services/
|
||||
├── 📋 README.md # This file
|
||||
├── 🎛️ chorus.sh # Management script
|
||||
├── 🐳 docker-compose.yml # Service orchestration
|
||||
├── 🗄️ init-db.sql # Database initialization
|
||||
├── 📊 monitoring/ # Monitoring configuration
|
||||
│ ├── prometheus.yml
|
||||
│ └── grafana/
|
||||
├── 📚 docs/ # Project documentation
|
||||
│ ├── PROJECT_PLAN.md
|
||||
│ ├── DESIGN.md
|
||||
│ └── Copywriting.md
|
||||
└── 🧩 modules/ # Git submodules
|
||||
├── whoosh/ # Orchestration platform
|
||||
├── bzzz/ # P2P coordination
|
||||
└── slurp/ # Context management (HCFS)
|
||||
```
|
||||
|
||||
## ⚙️ Configuration
|
||||
|
||||
### Environment Variables
|
||||
Key configuration options in `docker-compose.yml`:
|
||||
|
||||
- **Database**: `DATABASE_URL`, `POSTGRES_*`
|
||||
- **Redis**: `REDIS_URL`
|
||||
- **CORS**: `CORS_ORIGINS`
|
||||
- **Logging**: `LOG_LEVEL`
|
||||
- **Environment**: `ENVIRONMENT` (development/production)
|
||||
|
||||
### Adding New Agents
|
||||
Edit agent configurations in `modules/whoosh/config/whoosh.yaml`:
|
||||
|
||||
```yaml
|
||||
whoosh:
|
||||
agents:
|
||||
my_agent:
|
||||
name: "My Agent"
|
||||
endpoint: "http://192.168.1.100:11434"
|
||||
model: "llama3.1"
|
||||
specialization: "coding"
|
||||
capabilities: ["python", "javascript"]
|
||||
```
|
||||
|
||||
## 🔧 Development Setup
|
||||
|
||||
### Prerequisites
|
||||
- Docker and Docker Compose
|
||||
- Git
|
||||
- 8GB+ RAM recommended
|
||||
- Access to Ollama agents on network
|
||||
|
||||
### First-Time Setup
|
||||
```bash
|
||||
# Clone with submodules
|
||||
git clone --recursive <repository-url>
|
||||
cd chorus.services
|
||||
|
||||
# Or initialize submodules if already cloned
|
||||
./chorus.sh init
|
||||
|
||||
# Build all images
|
||||
./chorus.sh build
|
||||
|
||||
# Start services
|
||||
./chorus.sh start
|
||||
```
|
||||
|
||||
### Development Workflow
|
||||
```bash
|
||||
# Update submodules to latest
|
||||
./chorus.sh update
|
||||
|
||||
# View logs during development
|
||||
./chorus.sh logs
|
||||
|
||||
# Check service health
|
||||
./chorus.sh health
|
||||
|
||||
# Restart after changes
|
||||
./chorus.sh restart
|
||||
```
|
||||
|
||||
## 🚀 Git Submodules Guide
|
||||
|
||||
CHORUS Services uses git submodules to integrate independent components:
|
||||
|
||||
### Basic Submodule Commands
|
||||
```bash
|
||||
# Initialize submodules (first time)
|
||||
git submodule init
|
||||
git submodule update
|
||||
|
||||
# Update to latest commits
|
||||
git submodule update --remote
|
||||
|
||||
# Check submodule status
|
||||
git submodule status
|
||||
|
||||
# Enter a submodule to work on it
|
||||
cd modules/whoosh
|
||||
git checkout main
|
||||
# Make changes, commit, push
|
||||
|
||||
# Return to main project and commit submodule updates
|
||||
cd ../..
|
||||
git add modules/whoosh
|
||||
git commit -m "Update whoosh submodule"
|
||||
```
|
||||
|
||||
### Working with Submodules
|
||||
- **Each submodule** is an independent git repository
|
||||
- **Changes within submodules** must be committed in the submodule first
|
||||
- **Parent project** tracks specific commits of each submodule
|
||||
- **Use `./chorus.sh update`** to pull latest changes from all submodules
|
||||
|
||||
## 📊 Monitoring & Metrics
|
||||
|
||||
### Key Metrics Tracked
|
||||
- **Agent Performance**: Response time, throughput, availability
|
||||
- **Context Management**: Search performance, storage efficiency
|
||||
- **P2P Network**: Peer connectivity, message latency
|
||||
- **System Health**: CPU, memory, GPU utilization
|
||||
- **Workflow Execution**: Success rate, execution time
|
||||
|
||||
### Grafana Dashboards
|
||||
- **CHORUS Overview**: Platform-wide health and metrics
|
||||
- **Agent Performance**: Individual agent monitoring
|
||||
- **Context Analytics**: SLURP usage and performance
|
||||
- **Network Health**: BZZZ P2P network monitoring
|
||||
|
||||
## 🛡️ Security Features
|
||||
|
||||
- **Authentication**: JWT tokens and API key support
|
||||
- **Role-based Access**: Context filtering by agent roles
|
||||
- **Audit Trails**: Complete logging of all operations
|
||||
- **Network Security**: Internal container networking
|
||||
- **Data Privacy**: Encrypted context storage
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
### Component Development
|
||||
Each component can be developed independently:
|
||||
|
||||
```bash
|
||||
# Work on WHOOSH orchestrator
|
||||
cd modules/whoosh
|
||||
# Follow component-specific development guide
|
||||
|
||||
# Work on BZZZ P2P system
|
||||
cd modules/bzzz
|
||||
# Follow Go development practices
|
||||
|
||||
# Work on SLURP context system
|
||||
cd modules/slurp
|
||||
# Follow Python development practices
|
||||
```
|
||||
|
||||
### Integration Testing
|
||||
```bash
|
||||
# Test full platform integration
|
||||
./chorus.sh start
|
||||
./chorus.sh health
|
||||
|
||||
# Run component-specific tests
|
||||
cd modules/[component]
|
||||
# Follow component test procedures
|
||||
```
|
||||
|
||||
## 📈 Performance Metrics
|
||||
|
||||
**Production-Ready Performance:**
|
||||
- **API Response Times**: <5ms cached, <50ms uncached
|
||||
- **Context Search**: <100ms semantic search across 1000+ contexts
|
||||
- **P2P Network**: <10ms peer communication latency
|
||||
- **Workflow Execution**: Support for complex multi-agent workflows
|
||||
- **Concurrent Agents**: Scales to 10+ simultaneous agents
|
||||
|
||||
## 🎯 Use Cases
|
||||
|
||||
### Enterprise AI Development
|
||||
- **Multi-agent software development** with specialized AI agents
|
||||
- **Context-aware code generation** with organizational knowledge
|
||||
- **Distributed task execution** across development infrastructure
|
||||
|
||||
### Research & Collaboration
|
||||
- **AI agent coordination research** with real-world deployment
|
||||
- **Context management studies** with hierarchical storage
|
||||
- **Distributed systems research** with P2P networking
|
||||
|
||||
### Production AI Systems
|
||||
- **Enterprise AI orchestration** with monitoring and compliance
|
||||
- **Context-aware AI applications** with persistent memory
|
||||
- **Scalable multi-agent deployments** with automatic coordination
|
||||
|
||||
## 📞 Support & Documentation
|
||||
|
||||
- **🛠️ Management**: `./chorus.sh` for all operations
|
||||
- **📋 Component Docs**: See individual `modules/*/README.md`
|
||||
- **🔧 API Documentation**: http://localhost:8087/docs (when running)
|
||||
- **📊 Monitoring**: http://localhost:3002 (Grafana dashboards)
|
||||
|
||||
---
|
||||
|
||||
## 🎉 Welcome to CHORUS Services!
|
||||
|
||||
**CHORUS Services represents the future of distributed AI orchestration**, providing enterprise-ready tools for context management, agent coordination, and intelligent workflow execution.
|
||||
|
||||
🎵 *"Individual components make music, but CHORUS Services creates symphony."*
|
||||
|
||||
**Ready to orchestrate your AI agents?**
|
||||
```bash
|
||||
./chorus.sh start
|
||||
```
|
||||
Reference in New Issue
Block a user