chorus-services/README.md

🎵 CHORUS Services - Distributed AI Orchestration Platform

CHORUS Services eliminates context loss, reduces hallucinations, and enables scalable multi-agent collaboration through intelligent context management and distributed reasoning.

🚀 Quick Start

# Initialize submodules (first time only)
./chorus.sh init

# Login to Docker registry
./chorus.sh login

# Build and push images to registry
./chorus.sh build

# Start all services
./chorus.sh start

# Check service health
./chorus.sh health

Access Points:

• 🌐 Marketing Website: https://www.chorus.services (production)
• 🎛️ Dashboard: https://dashboard.chorus.services (production)
• 📡 API: https://api.chorus.services (production)
• 📊 Grafana Monitoring: http://localhost:3002 (admin/chorusadmin)
• 🔍 Prometheus Metrics: http://localhost:9092

Local Development:

• Dashboard: http://localhost:3001
• API Docs: http://localhost:8087/docs

🏗️ Architecture Overview

CHORUS Services integrates five core components into a unified platform:

┌─────────────────────────────────────────────────────────────────┐
│                     CHORUS Services Platform                     │
├─────────────────────────────────────────────────────────────────┤
│  🎛️  WHOOSH Orchestrator    │  📊  Monitoring & Analytics      │
│  - Workflow Management       │  - Prometheus Metrics            │
│  - Agent Coordination        │  - Grafana Dashboards            │  
│  - Task Distribution         │  - Real-time Health Monitoring   │
├─────────────────────────────────────────────────────────────────┤
│  🐝 BZZZ P2P Network        │  🧠 SLURP Context Management     │
│  - Agent Mesh Networking    │  - Hierarchical Context Storage  │
│  - Peer Discovery           │  - Semantic Search & Indexing    │
│  - Distributed Coordination │  - Multi-language SDK Support    │
├─────────────────────────────────────────────────────────────────┤
│  🎯 COOEE Feedback System (RL Context SLURP Integration)       │
│  - Performance-based Learning  │  - Context Relevance Tuning   │
│  - Agent Feedback Collection   │  - Role-based Access Control  │
└─────────────────────────────────────────────────────────────────┘

🧩 Core Components

🎛️ WHOOSH - Orchestration Engine

• Enterprise workflow management for AI agents
• Visual workflow editor with React Flow
• Real-time performance monitoring and metrics
• Multi-agent task distribution and coordination

🐝 BZZZ - P2P Agent Coordination

• Mesh networking with libp2p for resilient communication
• Automatic peer discovery via mDNS
• Distributed task coordination without single points of failure
• Go-based high-performance networking layer

🧠 SLURP - Context Curator Service

• Context curation from Hypercore logs based on agent roles and triggers
• Role-based context filtering for permissions, deprecation, feature changes
• SQL-based context delivery with intelligent relevance scoring
• Integration with HCFS for transparent filesystem-based context access

🎯 COOEE - Feedback & Learning (RL Context SLURP)

• Reinforcement learning for context relevance tuning
• Agent feedback collection with upvote/downvote systems
• Role-based context filtering and access control
• Continuous improvement through real-world performance data

📊 Monitoring & Analytics

• Prometheus metrics collection across all services
• Grafana dashboards for visualization and alerting
• Health checks and performance monitoring
• Audit trails with complete traceability

🛠️ Management Commands

The ./chorus.sh script provides unified management:

# Service Management
./chorus.sh start          # Start all services (registry images)
./chorus.sh stop           # Stop all services  
./chorus.sh restart        # Restart all services
./chorus.sh status         # Show service status
./chorus.sh dev            # Start in development mode (local builds)

# Docker Registry Operations  
./chorus.sh login          # Login to Docker registry
./chorus.sh build          # Build and push all images to registry
./chorus.sh pull           # Pull latest images from registry

# Production Deployment
./chorus.sh deploy         # Deploy to Docker Swarm (production)
./chorus.sh undeploy       # Remove from Docker Swarm

# Development & Maintenance
./chorus.sh update         # Update submodules to latest
./chorus.sh logs [service] # View logs
./chorus.sh health         # Check service health
./chorus.sh clean          # Clean up resources

# First-time Setup
./chorus.sh init           # Initialize git submodules

🌐 Service Endpoints

Service	Port	Purpose	Health Check
WHOOSH Dashboard	3001	Web UI for orchestration	http://localhost:3001
WHOOSH API	8087	REST API + WebSocket	http://localhost:8087/health
BZZZ Coordinator	8080	P2P coordination API	http://localhost:8080/health
SLURP API	8088	Context management API	http://localhost:8088/health
COOEE RL Tuner	8089	Feedback & learning API	http://localhost:8089/health
Grafana	3002	Monitoring dashboards	http://localhost:3002
Prometheus	9092	Metrics collection	http://localhost:9092
PostgreSQL	5433	Database	-
Redis	6380	Cache & message queue	-

📁 Project Structure

chorus.services/
├── 📋 README.md                    # This file
├── 🎛️ chorus.sh                   # Management script
├── 🐳 docker-compose.yml          # Service orchestration
├── 🗄️ init-db.sql                 # Database initialization
├── 📊 monitoring/                  # Monitoring configuration
│   ├── prometheus.yml
│   └── grafana/
├── 📚 docs/                        # Project documentation
│   ├── PROJECT_PLAN.md
│   ├── DESIGN.md
│   └── Copywriting.md
└── 🧩 modules/                     # Git submodules
    ├── whoosh/                     # Orchestration platform
    ├── bzzz/                       # P2P coordination
    └── slurp/                      # Context management (HCFS)

⚙️ Configuration

Environment Variables

Key configuration options in docker-compose.yml:

• Database: DATABASE_URL, POSTGRES_*
• Redis: REDIS_URL
• CORS: CORS_ORIGINS
• Logging: LOG_LEVEL
• Environment: ENVIRONMENT (development/production)

Adding New Agents

Edit agent configurations in modules/whoosh/config/whoosh.yaml:

whoosh:
  agents:
    my_agent:
      name: "My Agent"
      endpoint: "http://192.168.1.100:11434"
      model: "llama3.1"
      specialization: "coding"
      capabilities: ["python", "javascript"]

🔧 Development Setup

Prerequisites

• Docker and Docker Compose
• Git
• 8GB+ RAM recommended
• Access to Ollama agents on network

First-Time Setup

# Clone with submodules
git clone --recursive <repository-url>
cd chorus.services

# Or initialize submodules if already cloned
./chorus.sh init

# Login to Docker registry
./chorus.sh login

# Build and push all images to registry
./chorus.sh build

# Start services
./chorus.sh start

Development Workflow

# Development mode (local builds, live reloading)
./chorus.sh dev

# Update submodules to latest
./chorus.sh update

# Rebuild and push after changes
./chorus.sh build

# Pull latest images from registry
./chorus.sh pull

# View logs during development
./chorus.sh logs

# Check service health
./chorus.sh health

# Restart after changes
./chorus.sh restart

Production Deployment

# Deploy to Docker Swarm (production)
./chorus.sh deploy

# Access at https://*.home.deepblack.cloud endpoints

# Remove from swarm
./chorus.sh undeploy

🚀 Git Submodules Guide

CHORUS Services uses git submodules to integrate independent components:

Basic Submodule Commands

# Initialize submodules (first time)
git submodule init
git submodule update

# Update to latest commits
git submodule update --remote

# Check submodule status
git submodule status

# Enter a submodule to work on it
cd modules/whoosh
git checkout main
# Make changes, commit, push

# Return to main project and commit submodule updates
cd ../..
git add modules/whoosh
git commit -m "Update whoosh submodule"

Working with Submodules

• Each submodule is an independent git repository
• Changes within submodules must be committed in the submodule first
• Parent project tracks specific commits of each submodule
• Use ./chorus.sh update to pull latest changes from all submodules

📊 Monitoring & Metrics

Key Metrics Tracked

• Agent Performance: Response time, throughput, availability
• Context Management: Search performance, storage efficiency
• P2P Network: Peer connectivity, message latency
• System Health: CPU, memory, GPU utilization
• Workflow Execution: Success rate, execution time

Grafana Dashboards

• CHORUS Overview: Platform-wide health and metrics
• Agent Performance: Individual agent monitoring
• Context Analytics: SLURP usage and performance
• Network Health: BZZZ P2P network monitoring

🛡️ Security Features

• Authentication: JWT tokens and API key support
• Role-based Access: Context filtering by agent roles
• Audit Trails: Complete logging of all operations
• Network Security: Internal container networking
• Data Privacy: Encrypted context storage

🤝 Contributing

Component Development

Each component can be developed independently:

# Work on WHOOSH orchestrator
cd modules/whoosh
# Follow component-specific development guide

# Work on BZZZ P2P system  
cd modules/bzzz
# Follow Go development practices

# Work on SLURP context system
cd modules/slurp
# Follow Python development practices

Integration Testing

# Test full platform integration
./chorus.sh start
./chorus.sh health

# Run component-specific tests
cd modules/[component]
# Follow component test procedures

📈 Performance Metrics

Production-Ready Performance:

• API Response Times: <5ms cached, <50ms uncached
• Context Search: <100ms semantic search across 1000+ contexts
• P2P Network: <10ms peer communication latency
• Workflow Execution: Support for complex multi-agent workflows
• Concurrent Agents: Scales to 10+ simultaneous agents

🎯 Use Cases

Enterprise AI Development

• Multi-agent software development with specialized AI agents
• Context-aware code generation with organizational knowledge
• Distributed task execution across development infrastructure

Research & Collaboration

• AI agent coordination research with real-world deployment
• Context management studies with hierarchical storage
• Distributed systems research with P2P networking

Production AI Systems

• Enterprise AI orchestration with monitoring and compliance
• Context-aware AI applications with persistent memory
• Scalable multi-agent deployments with automatic coordination

📞 Support & Documentation

• 🛠️ Management: ./chorus.sh for all operations
• 📋 Component Docs: See individual modules/*/README.md
• 🔧 API Documentation: http://localhost:8087/docs (when running)
• 📊 Monitoring: http://localhost:3002 (Grafana dashboards)

---

🎉 Welcome to CHORUS Services!

CHORUS Services represents the future of distributed AI orchestration, providing enterprise-ready tools for context management, agent coordination, and intelligent workflow execution.

🎵 "Individual components make music, but CHORUS Services creates symphony."

Ready to orchestrate your AI agents?

./chorus.sh start