chorus-services/README.md

# 🎵 CHORUS Services - Distributed AI Orchestration Platform

**CHORUS Services** eliminates context loss, reduces hallucinations, and enables scalable multi-agent collaboration through intelligent context management and distributed reasoning.

## 🚀 Quick Start

```bash
# Initialize submodules (first time only)
./chorus.sh init

# Login to Docker registry
./chorus.sh login

# Build and push images to registry
./chorus.sh build

# Start all services
./chorus.sh start

# Check service health
./chorus.sh health
```

**Access Points:**
- 🌐 **Marketing Website**: https://www.chorus.services (production)
- 🎛️ **Dashboard**: https://dashboard.chorus.services (production)
- 📡 **API**: https://api.chorus.services (production)
- 📊 **Grafana Monitoring**: http://localhost:3002 (admin/chorusadmin)
- 🔍 **Prometheus Metrics**: http://localhost:9092

**Local Development:**
- Dashboard: http://localhost:3001
- API Docs: http://localhost:8087/docs

## 🏗️ Architecture Overview

CHORUS Services integrates five core components into a unified platform:

```
┌─────────────────────────────────────────────────────────────────┐
│                     CHORUS Services Platform                     │
├─────────────────────────────────────────────────────────────────┤
│  🎛️  WHOOSH Orchestrator    │  📊  Monitoring & Analytics      │
│  - Workflow Management       │  - Prometheus Metrics            │
│  - Agent Coordination        │  - Grafana Dashboards            │  
│  - Task Distribution         │  - Real-time Health Monitoring   │
├─────────────────────────────────────────────────────────────────┤
│  🐝 BZZZ P2P Network        │  🧠 SLURP Context Management     │
│  - Agent Mesh Networking    │  - Hierarchical Context Storage  │
│  - Peer Discovery           │  - Semantic Search & Indexing    │
│  - Distributed Coordination │  - Multi-language SDK Support    │
├─────────────────────────────────────────────────────────────────┤
│  🎯 COOEE Feedback System (RL Context SLURP Integration)       │
│  - Performance-based Learning  │  - Context Relevance Tuning   │
│  - Agent Feedback Collection   │  - Role-based Access Control  │
└─────────────────────────────────────────────────────────────────┘
```

## 🧩 Core Components

### 🎛️ WHOOSH - Orchestration Engine
- **Enterprise workflow management** for AI agents
- **Visual workflow editor** with React Flow
- **Real-time performance monitoring** and metrics
- **Multi-agent task distribution** and coordination

### 🐝 BZZZ - P2P Agent Coordination  
- **Mesh networking** with libp2p for resilient communication
- **Automatic peer discovery** via mDNS
- **Distributed task coordination** without single points of failure
- **Go-based** high-performance networking layer

### 🧠 SLURP - Context Curator Service
- **Context curation** from Hypercore logs based on agent roles and triggers
- **Role-based context filtering** for permissions, deprecation, feature changes
- **SQL-based context delivery** with intelligent relevance scoring
- **Integration with HCFS** for transparent filesystem-based context access

### 🎯 COOEE - Feedback & Learning (RL Context SLURP)
- **Reinforcement learning** for context relevance tuning
- **Agent feedback collection** with upvote/downvote systems
- **Role-based context filtering** and access control
- **Continuous improvement** through real-world performance data

### 📊 Monitoring & Analytics
- **Prometheus metrics collection** across all services
- **Grafana dashboards** for visualization and alerting
- **Health checks** and performance monitoring
- **Audit trails** with complete traceability

## 🛠️ Management Commands

The `./chorus.sh` script provides unified management:

```bash
# Service Management
./chorus.sh start          # Start all services (registry images)
./chorus.sh stop           # Stop all services  
./chorus.sh restart        # Restart all services
./chorus.sh status         # Show service status
./chorus.sh dev            # Start in development mode (local builds)

# Docker Registry Operations  
./chorus.sh login          # Login to Docker registry
./chorus.sh build          # Build and push all images to registry
./chorus.sh pull           # Pull latest images from registry

# Individual Component Builds
./build-and-push.sh website        # Build and push website only
./build-and-push.sh whoosh-backend # Build and push WHOOSH backend only
./build-and-push.sh bzzz           # Build and push BZZZ coordinator only

# Production Deployment
./chorus.sh deploy         # Deploy to Docker Swarm (production)
./chorus.sh undeploy       # Remove from Docker Swarm

# Development & Maintenance
./chorus.sh update         # Update submodules to latest
./chorus.sh logs [service] # View logs
./chorus.sh health         # Check service health
./chorus.sh clean          # Clean up resources

# First-time Setup
./chorus.sh init           # Initialize git submodules
```

## 🌐 Service Endpoints

| Service | Port | Purpose | Health Check |
|---------|------|---------|--------------|
| **WHOOSH Dashboard** | 3001 | Web UI for orchestration | http://localhost:3001 |
| **WHOOSH API** | 8087 | REST API + WebSocket | http://localhost:8087/health |
| **BZZZ Coordinator** | 8080 | P2P coordination API | http://localhost:8080/health |
| **SLURP API** | 8088 | Context management API | http://localhost:8088/health |
| **COOEE RL Tuner** | 8089 | Feedback & learning API | http://localhost:8089/health |
| **Grafana** | 3002 | Monitoring dashboards | http://localhost:3002 |
| **Prometheus** | 9092 | Metrics collection | http://localhost:9092 |
| **PostgreSQL** | 5433 | Database | - |
| **Redis** | 6380 | Cache & message queue | - |

## 📁 Project Structure

```
chorus.services/
├── 📋 README.md                    # This file
├── 🎛️ chorus.sh                   # Management script
├── 🐳 docker-compose.yml          # Service orchestration
├── 🗄️ init-db.sql                 # Database initialization
├── 📊 monitoring/                  # Monitoring configuration
│   ├── prometheus.yml
│   └── grafana/
├── 📚 docs/                        # Project documentation
│   ├── PROJECT_PLAN.md
│   ├── DESIGN.md
│   └── Copywriting.md
└── 🧩 modules/                     # Git submodules
    ├── whoosh/                     # Orchestration platform
    ├── bzzz/                       # P2P coordination
    ├── slurp/                      # Context management (HCFS)
    └── website/                    # Marketing website (Next.js)
```

## ⚙️ Configuration

### Environment Variables
Key configuration options in `docker-compose.yml`:

- **Database**: `DATABASE_URL`, `POSTGRES_*`
- **Redis**: `REDIS_URL`
- **CORS**: `CORS_ORIGINS`
- **Logging**: `LOG_LEVEL`
- **Environment**: `ENVIRONMENT` (development/production)

### Adding New Agents
Edit agent configurations in `modules/whoosh/config/whoosh.yaml`:

```yaml
whoosh:
  agents:
    my_agent:
      name: "My Agent"
      endpoint: "http://192.168.1.100:11434"
      model: "llama3.1"
      specialization: "coding"
      capabilities: ["python", "javascript"]
```

## 🔧 Development Setup

### Prerequisites
- Docker and Docker Compose
- Git
- 8GB+ RAM recommended
- Access to Ollama agents on network

### First-Time Setup
```bash
# Clone with submodules
git clone --recursive <repository-url>
cd chorus.services

# Or initialize submodules if already cloned
./chorus.sh init

# Login to Docker registry
./chorus.sh login

# Build and push all images to registry
./chorus.sh build

# Start services
./chorus.sh start
```

### Development Workflow
```bash
# Development mode (local builds, live reloading)
./chorus.sh dev

# Update submodules to latest
./chorus.sh update

# Rebuild and push after changes
./chorus.sh build

# Pull latest images from registry
./chorus.sh pull

# View logs during development
./chorus.sh logs

# Check service health
./chorus.sh health

# Restart after changes
./chorus.sh restart
```

### Production Deployment
```bash
# Deploy to Docker Swarm (production)
./chorus.sh deploy

# Access at https://*.home.deepblack.cloud endpoints

# Remove from swarm
./chorus.sh undeploy
```

## 🚀 Git Submodules Guide

CHORUS Services uses git submodules to integrate independent components:

### Basic Submodule Commands
```bash
# Initialize submodules (first time)
git submodule init
git submodule update

# Update to latest commits
git submodule update --remote

# Check submodule status
git submodule status

# Enter a submodule to work on it
cd modules/whoosh
git checkout main
# Make changes, commit, push

# Return to main project and commit submodule updates
cd ../..
git add modules/whoosh
git commit -m "Update whoosh submodule"
```

### Working with Submodules
- **Each submodule** is an independent git repository
- **Changes within submodules** must be committed in the submodule first
- **Parent project** tracks specific commits of each submodule
- **Use `./chorus.sh update`** to pull latest changes from all submodules

## 📊 Monitoring & Metrics

### Key Metrics Tracked
- **Agent Performance**: Response time, throughput, availability
- **Context Management**: Search performance, storage efficiency  
- **P2P Network**: Peer connectivity, message latency
- **System Health**: CPU, memory, GPU utilization
- **Workflow Execution**: Success rate, execution time

### Grafana Dashboards
- **CHORUS Overview**: Platform-wide health and metrics
- **Agent Performance**: Individual agent monitoring
- **Context Analytics**: SLURP usage and performance
- **Network Health**: BZZZ P2P network monitoring

## 🛡️ Security Features

- **Authentication**: JWT tokens and API key support
- **Role-based Access**: Context filtering by agent roles
- **Audit Trails**: Complete logging of all operations
- **Network Security**: Internal container networking
- **Data Privacy**: Encrypted context storage

## 🤝 Contributing

### Component Development
Each component can be developed independently:

```bash
# Work on WHOOSH orchestrator
cd modules/whoosh
# Follow component-specific development guide

# Work on BZZZ P2P system  
cd modules/bzzz
# Follow Go development practices

# Work on SLURP context system
cd modules/slurp
# Follow Python development practices
```

### Integration Testing
```bash
# Test full platform integration
./chorus.sh start
./chorus.sh health

# Run component-specific tests
cd modules/[component]
# Follow component test procedures
```

## 📈 Performance Metrics

**Production-Ready Performance:**
- **API Response Times**: <5ms cached, <50ms uncached
- **Context Search**: <100ms semantic search across 1000+ contexts
- **P2P Network**: <10ms peer communication latency
- **Workflow Execution**: Support for complex multi-agent workflows
- **Concurrent Agents**: Scales to 10+ simultaneous agents

## 🎯 Use Cases

### Enterprise AI Development
- **Multi-agent software development** with specialized AI agents
- **Context-aware code generation** with organizational knowledge
- **Distributed task execution** across development infrastructure

### Research & Collaboration  
- **AI agent coordination research** with real-world deployment
- **Context management studies** with hierarchical storage
- **Distributed systems research** with P2P networking

### Production AI Systems
- **Enterprise AI orchestration** with monitoring and compliance
- **Context-aware AI applications** with persistent memory
- **Scalable multi-agent deployments** with automatic coordination

## 📞 Support & Documentation

- **🛠️ Management**: `./chorus.sh` for all operations
- **📋 Component Docs**: See individual `modules/*/README.md`
- **🔧 API Documentation**: http://localhost:8087/docs (when running)
- **📊 Monitoring**: http://localhost:3002 (Grafana dashboards)

---

## 🎉 Welcome to CHORUS Services!

**CHORUS Services represents the future of distributed AI orchestration**, providing enterprise-ready tools for context management, agent coordination, and intelligent workflow execution.

🎵 *"Individual components make music, but CHORUS Services creates symphony."*

**Ready to orchestrate your AI agents?**
```bash
./chorus.sh start
```