b45c9c537e
Major architectural correction based on user clarification: ARCHITECTURE CHANGES: - SLURP: Context curator service that processes Hypercore logs, filters by role/triggers, serves via SQL - HCFS: FUSE filesystem overlay embedded in sandbox Docker images, not a separate service - Agents access context transparently through filesystem operations via HCFS → SLURP DOCKER CONFIGURATION: - Renamed slurp-api → slurp-curator throughout all configurations - Created proper Dockerfiles for SLURP curator and COOEE RL tuner services - Updated environment variables to reflect curator service role - Fixed missing slurp_data volume in docker-compose.yml UPDATED FILES: - README.md: Corrected SLURP description from "Context Management" to "Context Curator Service" - docker-compose.yml: Service rename, environment updates, added missing volume - docker-compose.swarm.yml: Production deployment with correct service names - build-and-push.sh: Updated to build slurp-curator service - chorus.sh: Updated service references and endpoints - init-db.sql: Database initialization for curator service - modules/slurp/Dockerfile: New curator service container - modules/slurp/Dockerfile.rl-tuner: New COOEE RL tuner container VALIDATION: - WHOOSH submodule validated: proper Dockerfiles and dependencies present - BZZZ submodule integrated: build issues identified in original repo (requires separate fix) - All Docker configurations updated to reflect correct architecture MEMORY UPDATES: - project_overview: Updated with correct SLURP/HCFS roles and current project status - Removed references to deprecated HMMM component This commit ensures all configuration reflects the correct architectural understanding where SLURP curates context from logs and HCFS provides transparent filesystem access to that curated context within agent sandbox environments. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
374 lines
12 KiB
Markdown
374 lines
12 KiB
Markdown
# 🎵 CHORUS Services - Distributed AI Orchestration Platform
|
||
|
||
**CHORUS Services** eliminates context loss, reduces hallucinations, and enables scalable multi-agent collaboration through intelligent context management and distributed reasoning.
|
||
|
||
## 🚀 Quick Start
|
||
|
||
```bash
|
||
# Initialize submodules (first time only)
|
||
./chorus.sh init
|
||
|
||
# Login to Docker registry
|
||
./chorus.sh login
|
||
|
||
# Build and push images to registry
|
||
./chorus.sh build
|
||
|
||
# Start all services
|
||
./chorus.sh start
|
||
|
||
# Check service health
|
||
./chorus.sh health
|
||
```
|
||
|
||
**Access Points:**
|
||
- 🌐 **Marketing Website**: https://www.chorus.services (production)
|
||
- 🎛️ **Dashboard**: https://dashboard.chorus.services (production)
|
||
- 📡 **API**: https://api.chorus.services (production)
|
||
- 📊 **Grafana Monitoring**: http://localhost:3002 (admin/chorusadmin)
|
||
- 🔍 **Prometheus Metrics**: http://localhost:9092
|
||
|
||
**Local Development:**
|
||
- Dashboard: http://localhost:3001
|
||
- API Docs: http://localhost:8087/docs
|
||
|
||
## 🏗️ Architecture Overview
|
||
|
||
CHORUS Services integrates five core components into a unified platform:
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ CHORUS Services Platform │
|
||
├─────────────────────────────────────────────────────────────────┤
|
||
│ 🎛️ WHOOSH Orchestrator │ 📊 Monitoring & Analytics │
|
||
│ - Workflow Management │ - Prometheus Metrics │
|
||
│ - Agent Coordination │ - Grafana Dashboards │
|
||
│ - Task Distribution │ - Real-time Health Monitoring │
|
||
├─────────────────────────────────────────────────────────────────┤
|
||
│ 🐝 BZZZ P2P Network │ 🧠 SLURP Context Management │
|
||
│ - Agent Mesh Networking │ - Hierarchical Context Storage │
|
||
│ - Peer Discovery │ - Semantic Search & Indexing │
|
||
│ - Distributed Coordination │ - Multi-language SDK Support │
|
||
├─────────────────────────────────────────────────────────────────┤
|
||
│ 🎯 COOEE Feedback System (RL Context SLURP Integration) │
|
||
│ - Performance-based Learning │ - Context Relevance Tuning │
|
||
│ - Agent Feedback Collection │ - Role-based Access Control │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## 🧩 Core Components
|
||
|
||
### 🎛️ WHOOSH - Orchestration Engine
|
||
- **Enterprise workflow management** for AI agents
|
||
- **Visual workflow editor** with React Flow
|
||
- **Real-time performance monitoring** and metrics
|
||
- **Multi-agent task distribution** and coordination
|
||
|
||
### 🐝 BZZZ - P2P Agent Coordination
|
||
- **Mesh networking** with libp2p for resilient communication
|
||
- **Automatic peer discovery** via mDNS
|
||
- **Distributed task coordination** without single points of failure
|
||
- **Go-based** high-performance networking layer
|
||
|
||
### 🧠 SLURP - Context Curator Service
|
||
- **Context curation** from Hypercore logs based on agent roles and triggers
|
||
- **Role-based context filtering** for permissions, deprecation, feature changes
|
||
- **SQL-based context delivery** with intelligent relevance scoring
|
||
- **Integration with HCFS** for transparent filesystem-based context access
|
||
|
||
### 🎯 COOEE - Feedback & Learning (RL Context SLURP)
|
||
- **Reinforcement learning** for context relevance tuning
|
||
- **Agent feedback collection** with upvote/downvote systems
|
||
- **Role-based context filtering** and access control
|
||
- **Continuous improvement** through real-world performance data
|
||
|
||
### 📊 Monitoring & Analytics
|
||
- **Prometheus metrics collection** across all services
|
||
- **Grafana dashboards** for visualization and alerting
|
||
- **Health checks** and performance monitoring
|
||
- **Audit trails** with complete traceability
|
||
|
||
## 🛠️ Management Commands
|
||
|
||
The `./chorus.sh` script provides unified management:
|
||
|
||
```bash
|
||
# Service Management
|
||
./chorus.sh start # Start all services (registry images)
|
||
./chorus.sh stop # Stop all services
|
||
./chorus.sh restart # Restart all services
|
||
./chorus.sh status # Show service status
|
||
./chorus.sh dev # Start in development mode (local builds)
|
||
|
||
# Docker Registry Operations
|
||
./chorus.sh login # Login to Docker registry
|
||
./chorus.sh build # Build and push all images to registry
|
||
./chorus.sh pull # Pull latest images from registry
|
||
|
||
# Production Deployment
|
||
./chorus.sh deploy # Deploy to Docker Swarm (production)
|
||
./chorus.sh undeploy # Remove from Docker Swarm
|
||
|
||
# Development & Maintenance
|
||
./chorus.sh update # Update submodules to latest
|
||
./chorus.sh logs [service] # View logs
|
||
./chorus.sh health # Check service health
|
||
./chorus.sh clean # Clean up resources
|
||
|
||
# First-time Setup
|
||
./chorus.sh init # Initialize git submodules
|
||
```
|
||
|
||
## 🌐 Service Endpoints
|
||
|
||
| Service | Port | Purpose | Health Check |
|
||
|---------|------|---------|--------------|
|
||
| **WHOOSH Dashboard** | 3001 | Web UI for orchestration | http://localhost:3001 |
|
||
| **WHOOSH API** | 8087 | REST API + WebSocket | http://localhost:8087/health |
|
||
| **BZZZ Coordinator** | 8080 | P2P coordination API | http://localhost:8080/health |
|
||
| **SLURP API** | 8088 | Context management API | http://localhost:8088/health |
|
||
| **COOEE RL Tuner** | 8089 | Feedback & learning API | http://localhost:8089/health |
|
||
| **Grafana** | 3002 | Monitoring dashboards | http://localhost:3002 |
|
||
| **Prometheus** | 9092 | Metrics collection | http://localhost:9092 |
|
||
| **PostgreSQL** | 5433 | Database | - |
|
||
| **Redis** | 6380 | Cache & message queue | - |
|
||
|
||
## 📁 Project Structure
|
||
|
||
```
|
||
chorus.services/
|
||
├── 📋 README.md # This file
|
||
├── 🎛️ chorus.sh # Management script
|
||
├── 🐳 docker-compose.yml # Service orchestration
|
||
├── 🗄️ init-db.sql # Database initialization
|
||
├── 📊 monitoring/ # Monitoring configuration
|
||
│ ├── prometheus.yml
|
||
│ └── grafana/
|
||
├── 📚 docs/ # Project documentation
|
||
│ ├── PROJECT_PLAN.md
|
||
│ ├── DESIGN.md
|
||
│ └── Copywriting.md
|
||
└── 🧩 modules/ # Git submodules
|
||
├── whoosh/ # Orchestration platform
|
||
├── bzzz/ # P2P coordination
|
||
└── slurp/ # Context management (HCFS)
|
||
```
|
||
|
||
## ⚙️ Configuration
|
||
|
||
### Environment Variables
|
||
Key configuration options in `docker-compose.yml`:
|
||
|
||
- **Database**: `DATABASE_URL`, `POSTGRES_*`
|
||
- **Redis**: `REDIS_URL`
|
||
- **CORS**: `CORS_ORIGINS`
|
||
- **Logging**: `LOG_LEVEL`
|
||
- **Environment**: `ENVIRONMENT` (development/production)
|
||
|
||
### Adding New Agents
|
||
Edit agent configurations in `modules/whoosh/config/whoosh.yaml`:
|
||
|
||
```yaml
|
||
whoosh:
|
||
agents:
|
||
my_agent:
|
||
name: "My Agent"
|
||
endpoint: "http://192.168.1.100:11434"
|
||
model: "llama3.1"
|
||
specialization: "coding"
|
||
capabilities: ["python", "javascript"]
|
||
```
|
||
|
||
## 🔧 Development Setup
|
||
|
||
### Prerequisites
|
||
- Docker and Docker Compose
|
||
- Git
|
||
- 8GB+ RAM recommended
|
||
- Access to Ollama agents on network
|
||
|
||
### First-Time Setup
|
||
```bash
|
||
# Clone with submodules
|
||
git clone --recursive <repository-url>
|
||
cd chorus.services
|
||
|
||
# Or initialize submodules if already cloned
|
||
./chorus.sh init
|
||
|
||
# Login to Docker registry
|
||
./chorus.sh login
|
||
|
||
# Build and push all images to registry
|
||
./chorus.sh build
|
||
|
||
# Start services
|
||
./chorus.sh start
|
||
```
|
||
|
||
### Development Workflow
|
||
```bash
|
||
# Development mode (local builds, live reloading)
|
||
./chorus.sh dev
|
||
|
||
# Update submodules to latest
|
||
./chorus.sh update
|
||
|
||
# Rebuild and push after changes
|
||
./chorus.sh build
|
||
|
||
# Pull latest images from registry
|
||
./chorus.sh pull
|
||
|
||
# View logs during development
|
||
./chorus.sh logs
|
||
|
||
# Check service health
|
||
./chorus.sh health
|
||
|
||
# Restart after changes
|
||
./chorus.sh restart
|
||
```
|
||
|
||
### Production Deployment
|
||
```bash
|
||
# Deploy to Docker Swarm (production)
|
||
./chorus.sh deploy
|
||
|
||
# Access at https://*.home.deepblack.cloud endpoints
|
||
|
||
# Remove from swarm
|
||
./chorus.sh undeploy
|
||
```
|
||
|
||
## 🚀 Git Submodules Guide
|
||
|
||
CHORUS Services uses git submodules to integrate independent components:
|
||
|
||
### Basic Submodule Commands
|
||
```bash
|
||
# Initialize submodules (first time)
|
||
git submodule init
|
||
git submodule update
|
||
|
||
# Update to latest commits
|
||
git submodule update --remote
|
||
|
||
# Check submodule status
|
||
git submodule status
|
||
|
||
# Enter a submodule to work on it
|
||
cd modules/whoosh
|
||
git checkout main
|
||
# Make changes, commit, push
|
||
|
||
# Return to main project and commit submodule updates
|
||
cd ../..
|
||
git add modules/whoosh
|
||
git commit -m "Update whoosh submodule"
|
||
```
|
||
|
||
### Working with Submodules
|
||
- **Each submodule** is an independent git repository
|
||
- **Changes within submodules** must be committed in the submodule first
|
||
- **Parent project** tracks specific commits of each submodule
|
||
- **Use `./chorus.sh update`** to pull latest changes from all submodules
|
||
|
||
## 📊 Monitoring & Metrics
|
||
|
||
### Key Metrics Tracked
|
||
- **Agent Performance**: Response time, throughput, availability
|
||
- **Context Management**: Search performance, storage efficiency
|
||
- **P2P Network**: Peer connectivity, message latency
|
||
- **System Health**: CPU, memory, GPU utilization
|
||
- **Workflow Execution**: Success rate, execution time
|
||
|
||
### Grafana Dashboards
|
||
- **CHORUS Overview**: Platform-wide health and metrics
|
||
- **Agent Performance**: Individual agent monitoring
|
||
- **Context Analytics**: SLURP usage and performance
|
||
- **Network Health**: BZZZ P2P network monitoring
|
||
|
||
## 🛡️ Security Features
|
||
|
||
- **Authentication**: JWT tokens and API key support
|
||
- **Role-based Access**: Context filtering by agent roles
|
||
- **Audit Trails**: Complete logging of all operations
|
||
- **Network Security**: Internal container networking
|
||
- **Data Privacy**: Encrypted context storage
|
||
|
||
## 🤝 Contributing
|
||
|
||
### Component Development
|
||
Each component can be developed independently:
|
||
|
||
```bash
|
||
# Work on WHOOSH orchestrator
|
||
cd modules/whoosh
|
||
# Follow component-specific development guide
|
||
|
||
# Work on BZZZ P2P system
|
||
cd modules/bzzz
|
||
# Follow Go development practices
|
||
|
||
# Work on SLURP context system
|
||
cd modules/slurp
|
||
# Follow Python development practices
|
||
```
|
||
|
||
### Integration Testing
|
||
```bash
|
||
# Test full platform integration
|
||
./chorus.sh start
|
||
./chorus.sh health
|
||
|
||
# Run component-specific tests
|
||
cd modules/[component]
|
||
# Follow component test procedures
|
||
```
|
||
|
||
## 📈 Performance Metrics
|
||
|
||
**Production-Ready Performance:**
|
||
- **API Response Times**: <5ms cached, <50ms uncached
|
||
- **Context Search**: <100ms semantic search across 1000+ contexts
|
||
- **P2P Network**: <10ms peer communication latency
|
||
- **Workflow Execution**: Support for complex multi-agent workflows
|
||
- **Concurrent Agents**: Scales to 10+ simultaneous agents
|
||
|
||
## 🎯 Use Cases
|
||
|
||
### Enterprise AI Development
|
||
- **Multi-agent software development** with specialized AI agents
|
||
- **Context-aware code generation** with organizational knowledge
|
||
- **Distributed task execution** across development infrastructure
|
||
|
||
### Research & Collaboration
|
||
- **AI agent coordination research** with real-world deployment
|
||
- **Context management studies** with hierarchical storage
|
||
- **Distributed systems research** with P2P networking
|
||
|
||
### Production AI Systems
|
||
- **Enterprise AI orchestration** with monitoring and compliance
|
||
- **Context-aware AI applications** with persistent memory
|
||
- **Scalable multi-agent deployments** with automatic coordination
|
||
|
||
## 📞 Support & Documentation
|
||
|
||
- **🛠️ Management**: `./chorus.sh` for all operations
|
||
- **📋 Component Docs**: See individual `modules/*/README.md`
|
||
- **🔧 API Documentation**: http://localhost:8087/docs (when running)
|
||
- **📊 Monitoring**: http://localhost:3002 (Grafana dashboards)
|
||
|
||
---
|
||
|
||
## 🎉 Welcome to CHORUS Services!
|
||
|
||
**CHORUS Services represents the future of distributed AI orchestration**, providing enterprise-ready tools for context management, agent coordination, and intelligent workflow execution.
|
||
|
||
🎵 *"Individual components make music, but CHORUS Services creates symphony."*
|
||
|
||
**Ready to orchestrate your AI agents?**
|
||
```bash
|
||
./chorus.sh start
|
||
``` |