diff --git a/README.md b/README.md new file mode 100644 index 0000000..b01b8ce --- /dev/null +++ b/README.md @@ -0,0 +1,331 @@ +# 🎵 CHORUS Services - Distributed AI Orchestration Platform + +**CHORUS Services** eliminates context loss, reduces hallucinations, and enables scalable multi-agent collaboration through intelligent context management and distributed reasoning. + +## 🚀 Quick Start + +```bash +# Initialize submodules (first time only) +./chorus.sh init + +# Start all services +./chorus.sh start + +# Check service health +./chorus.sh health +``` + +**Access Points:** +- 🌐 **WHOOSH Dashboard**: http://localhost:3001 +- 📡 **API Documentation**: http://localhost:8087/docs +- 📊 **Grafana Monitoring**: http://localhost:3002 (admin/chorusadmin) +- 🔍 **Prometheus Metrics**: http://localhost:9092 + +## 🏗️ Architecture Overview + +CHORUS Services integrates five core components into a unified platform: + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ CHORUS Services Platform │ +├─────────────────────────────────────────────────────────────────┤ +│ 🎛️ WHOOSH Orchestrator │ 📊 Monitoring & Analytics │ +│ - Workflow Management │ - Prometheus Metrics │ +│ - Agent Coordination │ - Grafana Dashboards │ +│ - Task Distribution │ - Real-time Health Monitoring │ +├─────────────────────────────────────────────────────────────────┤ +│ 🐝 BZZZ P2P Network │ 🧠 SLURP Context Management │ +│ - Agent Mesh Networking │ - Hierarchical Context Storage │ +│ - Peer Discovery │ - Semantic Search & Indexing │ +│ - Distributed Coordination │ - Multi-language SDK Support │ +├─────────────────────────────────────────────────────────────────┤ +│ 🎯 COOEE Feedback System (RL Context SLURP Integration) │ +│ - Performance-based Learning │ - Context Relevance Tuning │ +│ - Agent Feedback Collection │ - Role-based Access Control │ +└─────────────────────────────────────────────────────────────────┘ +``` + +## 🧩 Core Components + +### 🎛️ WHOOSH - Orchestration Engine +- **Enterprise workflow management** for AI agents +- **Visual workflow editor** with React Flow +- **Real-time performance monitoring** and metrics +- **Multi-agent task distribution** and coordination + +### 🐝 BZZZ - P2P Agent Coordination +- **Mesh networking** with libp2p for resilient communication +- **Automatic peer discovery** via mDNS +- **Distributed task coordination** without single points of failure +- **Go-based** high-performance networking layer + +### 🧠 SLURP - Context Management System +- **Hierarchical context storage** with path-based organization +- **Hybrid search** combining semantic similarity + BM25 +- **Multi-language SDKs** (Python, JavaScript, Go, Rust, Java, C#) +- **Enterprise-grade APIs** with authentication and WebSocket streaming + +### 🎯 COOEE - Feedback & Learning (RL Context SLURP) +- **Reinforcement learning** for context relevance tuning +- **Agent feedback collection** with upvote/downvote systems +- **Role-based context filtering** and access control +- **Continuous improvement** through real-world performance data + +### 📊 Monitoring & Analytics +- **Prometheus metrics collection** across all services +- **Grafana dashboards** for visualization and alerting +- **Health checks** and performance monitoring +- **Audit trails** with complete traceability + +## 🛠️ Management Commands + +The `./chorus.sh` script provides unified management: + +```bash +# Service Management +./chorus.sh start # Start all services +./chorus.sh stop # Stop all services +./chorus.sh restart # Restart all services +./chorus.sh status # Show service status + +# Development & Maintenance +./chorus.sh build # Build all docker images +./chorus.sh update # Update submodules to latest +./chorus.sh logs [service] # View logs +./chorus.sh health # Check service health +./chorus.sh clean # Clean up resources + +# First-time Setup +./chorus.sh init # Initialize git submodules +``` + +## 🌐 Service Endpoints + +| Service | Port | Purpose | Health Check | +|---------|------|---------|--------------| +| **WHOOSH Dashboard** | 3001 | Web UI for orchestration | http://localhost:3001 | +| **WHOOSH API** | 8087 | REST API + WebSocket | http://localhost:8087/health | +| **BZZZ Coordinator** | 8080 | P2P coordination API | http://localhost:8080/health | +| **SLURP API** | 8088 | Context management API | http://localhost:8088/health | +| **COOEE RL Tuner** | 8089 | Feedback & learning API | http://localhost:8089/health | +| **Grafana** | 3002 | Monitoring dashboards | http://localhost:3002 | +| **Prometheus** | 9092 | Metrics collection | http://localhost:9092 | +| **PostgreSQL** | 5433 | Database | - | +| **Redis** | 6380 | Cache & message queue | - | + +## 📁 Project Structure + +``` +chorus.services/ +├── 📋 README.md # This file +├── 🎛️ chorus.sh # Management script +├── 🐳 docker-compose.yml # Service orchestration +├── 🗄️ init-db.sql # Database initialization +├── 📊 monitoring/ # Monitoring configuration +│ ├── prometheus.yml +│ └── grafana/ +├── 📚 docs/ # Project documentation +│ ├── PROJECT_PLAN.md +│ ├── DESIGN.md +│ └── Copywriting.md +└── 🧩 modules/ # Git submodules + ├── whoosh/ # Orchestration platform + ├── bzzz/ # P2P coordination + └── slurp/ # Context management (HCFS) +``` + +## ⚙️ Configuration + +### Environment Variables +Key configuration options in `docker-compose.yml`: + +- **Database**: `DATABASE_URL`, `POSTGRES_*` +- **Redis**: `REDIS_URL` +- **CORS**: `CORS_ORIGINS` +- **Logging**: `LOG_LEVEL` +- **Environment**: `ENVIRONMENT` (development/production) + +### Adding New Agents +Edit agent configurations in `modules/whoosh/config/whoosh.yaml`: + +```yaml +whoosh: + agents: + my_agent: + name: "My Agent" + endpoint: "http://192.168.1.100:11434" + model: "llama3.1" + specialization: "coding" + capabilities: ["python", "javascript"] +``` + +## 🔧 Development Setup + +### Prerequisites +- Docker and Docker Compose +- Git +- 8GB+ RAM recommended +- Access to Ollama agents on network + +### First-Time Setup +```bash +# Clone with submodules +git clone --recursive +cd chorus.services + +# Or initialize submodules if already cloned +./chorus.sh init + +# Build all images +./chorus.sh build + +# Start services +./chorus.sh start +``` + +### Development Workflow +```bash +# Update submodules to latest +./chorus.sh update + +# View logs during development +./chorus.sh logs + +# Check service health +./chorus.sh health + +# Restart after changes +./chorus.sh restart +``` + +## 🚀 Git Submodules Guide + +CHORUS Services uses git submodules to integrate independent components: + +### Basic Submodule Commands +```bash +# Initialize submodules (first time) +git submodule init +git submodule update + +# Update to latest commits +git submodule update --remote + +# Check submodule status +git submodule status + +# Enter a submodule to work on it +cd modules/whoosh +git checkout main +# Make changes, commit, push + +# Return to main project and commit submodule updates +cd ../.. +git add modules/whoosh +git commit -m "Update whoosh submodule" +``` + +### Working with Submodules +- **Each submodule** is an independent git repository +- **Changes within submodules** must be committed in the submodule first +- **Parent project** tracks specific commits of each submodule +- **Use `./chorus.sh update`** to pull latest changes from all submodules + +## 📊 Monitoring & Metrics + +### Key Metrics Tracked +- **Agent Performance**: Response time, throughput, availability +- **Context Management**: Search performance, storage efficiency +- **P2P Network**: Peer connectivity, message latency +- **System Health**: CPU, memory, GPU utilization +- **Workflow Execution**: Success rate, execution time + +### Grafana Dashboards +- **CHORUS Overview**: Platform-wide health and metrics +- **Agent Performance**: Individual agent monitoring +- **Context Analytics**: SLURP usage and performance +- **Network Health**: BZZZ P2P network monitoring + +## 🛡️ Security Features + +- **Authentication**: JWT tokens and API key support +- **Role-based Access**: Context filtering by agent roles +- **Audit Trails**: Complete logging of all operations +- **Network Security**: Internal container networking +- **Data Privacy**: Encrypted context storage + +## 🤝 Contributing + +### Component Development +Each component can be developed independently: + +```bash +# Work on WHOOSH orchestrator +cd modules/whoosh +# Follow component-specific development guide + +# Work on BZZZ P2P system +cd modules/bzzz +# Follow Go development practices + +# Work on SLURP context system +cd modules/slurp +# Follow Python development practices +``` + +### Integration Testing +```bash +# Test full platform integration +./chorus.sh start +./chorus.sh health + +# Run component-specific tests +cd modules/[component] +# Follow component test procedures +``` + +## 📈 Performance Metrics + +**Production-Ready Performance:** +- **API Response Times**: <5ms cached, <50ms uncached +- **Context Search**: <100ms semantic search across 1000+ contexts +- **P2P Network**: <10ms peer communication latency +- **Workflow Execution**: Support for complex multi-agent workflows +- **Concurrent Agents**: Scales to 10+ simultaneous agents + +## 🎯 Use Cases + +### Enterprise AI Development +- **Multi-agent software development** with specialized AI agents +- **Context-aware code generation** with organizational knowledge +- **Distributed task execution** across development infrastructure + +### Research & Collaboration +- **AI agent coordination research** with real-world deployment +- **Context management studies** with hierarchical storage +- **Distributed systems research** with P2P networking + +### Production AI Systems +- **Enterprise AI orchestration** with monitoring and compliance +- **Context-aware AI applications** with persistent memory +- **Scalable multi-agent deployments** with automatic coordination + +## 📞 Support & Documentation + +- **🛠️ Management**: `./chorus.sh` for all operations +- **📋 Component Docs**: See individual `modules/*/README.md` +- **🔧 API Documentation**: http://localhost:8087/docs (when running) +- **📊 Monitoring**: http://localhost:3002 (Grafana dashboards) + +--- + +## 🎉 Welcome to CHORUS Services! + +**CHORUS Services represents the future of distributed AI orchestration**, providing enterprise-ready tools for context management, agent coordination, and intelligent workflow execution. + +🎵 *"Individual components make music, but CHORUS Services creates symphony."* + +**Ready to orchestrate your AI agents?** +```bash +./chorus.sh start +``` \ No newline at end of file diff --git a/chorus.sh b/chorus.sh new file mode 100755 index 0000000..8bf6654 --- /dev/null +++ b/chorus.sh @@ -0,0 +1,231 @@ +#!/bin/bash + +# CHORUS Services Management Script +# Unified management interface for all CHORUS Services components + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$SCRIPT_DIR" + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Print colored output +print_info() { + echo -e "${BLUE}[INFO]${NC} $1" +} + +print_success() { + echo -e "${GREEN}[SUCCESS]${NC} $1" +} + +print_warning() { + echo -e "${YELLOW}[WARNING]${NC} $1" +} + +print_error() { + echo -e "${RED}[ERROR]${NC} $1" +} + +# Show usage information +show_usage() { + echo "CHORUS Services Management Script" + echo "" + echo "Usage: $0 [COMMAND]" + echo "" + echo "Commands:" + echo " start Start all CHORUS Services" + echo " stop Stop all CHORUS Services" + echo " restart Restart all CHORUS Services" + echo " status Show status of all services" + echo " logs Show logs from all services" + echo " logs [service] Show logs from specific service" + echo " build Build all docker images" + echo " update Update all submodules to latest" + echo " init Initialize submodules (first time setup)" + echo " clean Clean up docker resources" + echo " health Check health of all services" + echo "" + echo "Services:" + echo " - whoosh-backend : Orchestration API (port 8087)" + echo " - whoosh-frontend : Web Dashboard (port 3001)" + echo " - bzzz-coordinator : P2P Coordination (port 8080)" + echo " - slurp-api : Context Management (port 8088)" + echo " - slurp-rl-tuner : RL Context Tuner (port 8089)" + echo " - postgres : Database (port 5433)" + echo " - redis : Cache (port 6380)" + echo " - prometheus : Metrics (port 9091)" + echo " - grafana : Monitoring (port 3002)" +} + +# Initialize git submodules +init_submodules() { + print_info "Initializing git submodules..." + git submodule init + git submodule update --recursive + print_success "Submodules initialized" +} + +# Update git submodules +update_submodules() { + print_info "Updating git submodules..." + git submodule update --remote --recursive + print_success "Submodules updated" +} + +# Build all docker images +build_images() { + print_info "Building CHORUS Services docker images..." + + # Build each service + print_info "Building WHOOSH backend..." + docker-compose build whoosh-backend + + print_info "Building WHOOSH frontend..." + docker-compose build whoosh-frontend + + print_info "Building BZZZ coordinator..." + docker-compose build bzzz-coordinator + + print_info "Building SLURP API..." + docker-compose build slurp-api + + print_info "Building SLURP RL Tuner..." + docker-compose build slurp-rl-tuner + + print_success "All images built successfully" +} + +# Start all services +start_services() { + print_info "Starting CHORUS Services..." + docker-compose up -d + + print_info "Waiting for services to start..." + sleep 10 + + print_success "CHORUS Services started" + print_info "Access points:" + echo " - WHOOSH Dashboard: http://localhost:3001" + echo " - WHOOSH API: http://localhost:8087" + echo " - BZZZ Coordinator: http://localhost:8080" + echo " - SLURP API: http://localhost:8088" + echo " - SLURP RL Tuner: http://localhost:8089" + echo " - Grafana: http://localhost:3002 (admin/chorusadmin)" + echo " - Prometheus: http://localhost:9092" +} + +# Stop all services +stop_services() { + print_info "Stopping CHORUS Services..." + docker-compose down + print_success "CHORUS Services stopped" +} + +# Restart all services +restart_services() { + print_info "Restarting CHORUS Services..." + stop_services + start_services +} + +# Show service status +show_status() { + print_info "CHORUS Services Status:" + docker-compose ps +} + +# Show logs +show_logs() { + if [ -z "$1" ]; then + print_info "Showing logs from all services..." + docker-compose logs -f + else + print_info "Showing logs from $1..." + docker-compose logs -f "$1" + fi +} + +# Check health of services +check_health() { + print_info "Checking CHORUS Services health..." + + services=( + "http://localhost:8087/health|WHOOSH Backend" + "http://localhost:3001|WHOOSH Frontend" + "http://localhost:8080/health|BZZZ Coordinator" + "http://localhost:8088/health|SLURP API" + "http://localhost:8089/health|SLURP RL Tuner" + "http://localhost:9092/-/healthy|Prometheus" + "http://localhost:3002/api/health|Grafana" + ) + + for service in "${services[@]}"; do + url=$(echo "$service" | cut -d'|' -f1) + name=$(echo "$service" | cut -d'|' -f2) + + if curl -s -f "$url" > /dev/null 2>&1; then + print_success "$name is healthy" + else + print_error "$name is not responding" + fi + done +} + +# Clean up docker resources +clean_resources() { + print_warning "This will remove all CHORUS Services containers, networks, and volumes" + read -p "Are you sure? (y/N): " -n 1 -r + echo + if [[ $REPLY =~ ^[Yy]$ ]]; then + print_info "Cleaning up docker resources..." + docker-compose down -v --remove-orphans + docker system prune -f + print_success "Cleanup completed" + else + print_info "Cleanup cancelled" + fi +} + +# Main command handler +case "$1" in + start) + start_services + ;; + stop) + stop_services + ;; + restart) + restart_services + ;; + status) + show_status + ;; + logs) + show_logs "$2" + ;; + build) + build_images + ;; + update) + update_submodules + ;; + init) + init_submodules + ;; + clean) + clean_resources + ;; + health) + check_health + ;; + *) + show_usage + exit 1 + ;; +esac \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..b11f57e --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,157 @@ +version: '3.8' + +services: + # WHOOSH - Orchestration Platform + whoosh-backend: + build: ./modules/whoosh/backend + container_name: chorus_whoosh_backend + ports: + - "8087:8000" + environment: + - DATABASE_URL=postgresql://chorus:choruspass@postgres:5432/chorus_whoosh + - REDIS_URL=redis://redis:6379 + - CORS_ORIGINS=http://localhost:3001,https://whoosh.home.deepblack.cloud + - ENVIRONMENT=development + depends_on: + - postgres + - redis + networks: + - chorus_network + volumes: + - ./modules/whoosh/backend:/app + - ./modules/whoosh/config:/app/config + + whoosh-frontend: + build: ./modules/whoosh/frontend + container_name: chorus_whoosh_frontend + ports: + - "3001:3000" + environment: + - REACT_APP_API_URL=http://localhost:8087 + - REACT_APP_WS_URL=ws://localhost:8087 + depends_on: + - whoosh-backend + networks: + - chorus_network + volumes: + - ./modules/whoosh/frontend:/app + - /app/node_modules + + # BZZZ - P2P Agent Coordination + bzzz-coordinator: + build: + context: ./modules/bzzz + dockerfile: Dockerfile + container_name: chorus_bzzz_coordinator + ports: + - "4001:4001" # libp2p port + - "8080:8080" # HTTP API port + environment: + - BZZZ_NODE_ENV=development + - BZZZ_LOG_LEVEL=info + networks: + - chorus_network + - host # Needed for P2P discovery + volumes: + - ./modules/bzzz/config:/app/config + - ./modules/bzzz/data:/app/data + + # SLURP - Context Management (HCFS) + slurp-api: + build: ./modules/slurp/hcfs-python + container_name: chorus_slurp_api + ports: + - "8088:8000" + environment: + - HCFS_DATABASE_URL=sqlite:///data/hcfs.db + - HCFS_LOG_LEVEL=info + - HCFS_AUTH_ENABLED=true + networks: + - chorus_network + volumes: + - ./modules/slurp/data:/app/data + - ./modules/slurp/config:/app/config + + # RL Context SLURP (COOEE equivalent) + slurp-rl-tuner: + build: + context: ./modules/slurp + dockerfile: Dockerfile.rl-tuner + container_name: chorus_slurp_rl_tuner + ports: + - "8089:8000" + environment: + - RL_TUNER_DATABASE_URL=postgresql://chorus:choruspass@postgres:5432/chorus_rl_tuner + - HCFS_API_URL=http://slurp-api:8000 + - BZZZ_API_URL=http://bzzz-coordinator:8080 + depends_on: + - postgres + - slurp-api + - bzzz-coordinator + networks: + - chorus_network + + # Shared Infrastructure + postgres: + image: postgres:15 + container_name: chorus_postgres + environment: + - POSTGRES_DB=chorus + - POSTGRES_USER=chorus + - POSTGRES_PASSWORD=choruspass + ports: + - "5433:5432" + volumes: + - postgres_data:/var/lib/postgresql/data + - ./init-db.sql:/docker-entrypoint-initdb.d/init-db.sql + networks: + - chorus_network + + redis: + image: redis:7-alpine + container_name: chorus_redis + ports: + - "6380:6379" + volumes: + - redis_data:/data + networks: + - chorus_network + + # Monitoring Stack + prometheus: + image: prom/prometheus:latest + container_name: chorus_prometheus + ports: + - "9092:9090" + volumes: + - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml + - prometheus_data:/prometheus + networks: + - chorus_network + + grafana: + image: grafana/grafana:latest + container_name: chorus_grafana + ports: + - "3002:3000" + environment: + - GF_SECURITY_ADMIN_PASSWORD=chorusadmin + volumes: + - grafana_data:/var/lib/grafana + - ./monitoring/grafana/dashboards:/etc/grafana/provisioning/dashboards + - ./monitoring/grafana/datasources:/etc/grafana/provisioning/datasources + networks: + - chorus_network + +volumes: + postgres_data: + redis_data: + prometheus_data: + grafana_data: + +networks: + chorus_network: + driver: bridge + ipam: + config: + - subnet: 172.20.0.0/16 \ No newline at end of file diff --git a/init-db.sql b/init-db.sql new file mode 100644 index 0000000..b7cfd29 --- /dev/null +++ b/init-db.sql @@ -0,0 +1,11 @@ +-- CHORUS Services Database Initialization + +-- Create databases for each service +CREATE DATABASE chorus_whoosh; +CREATE DATABASE chorus_rl_tuner; +CREATE DATABASE chorus_monitoring; + +-- Grant permissions +GRANT ALL PRIVILEGES ON DATABASE chorus_whoosh TO chorus; +GRANT ALL PRIVILEGES ON DATABASE chorus_rl_tuner TO chorus; +GRANT ALL PRIVILEGES ON DATABASE chorus_monitoring TO chorus; \ No newline at end of file diff --git a/monitoring/grafana/datasources/prometheus.yml b/monitoring/grafana/datasources/prometheus.yml new file mode 100644 index 0000000..8049912 --- /dev/null +++ b/monitoring/grafana/datasources/prometheus.yml @@ -0,0 +1,8 @@ +apiVersion: 1 + +datasources: + - name: Prometheus + type: prometheus + access: proxy + url: http://prometheus:9090 + isDefault: true \ No newline at end of file diff --git a/monitoring/prometheus.yml b/monitoring/prometheus.yml new file mode 100644 index 0000000..7c231f4 --- /dev/null +++ b/monitoring/prometheus.yml @@ -0,0 +1,32 @@ +global: + scrape_interval: 15s + evaluation_interval: 15s + +rule_files: + # - "first_rules.yml" + # - "second_rules.yml" + +scrape_configs: + - job_name: 'prometheus' + static_configs: + - targets: ['localhost:9090'] + + - job_name: 'whoosh-backend' + static_configs: + - targets: ['whoosh-backend:8000'] + metrics_path: '/metrics' + + - job_name: 'bzzz-coordinator' + static_configs: + - targets: ['bzzz-coordinator:8080'] + metrics_path: '/metrics' + + - job_name: 'slurp-api' + static_configs: + - targets: ['slurp-api:8000'] + metrics_path: '/metrics' + + - job_name: 'slurp-rl-tuner' + static_configs: + - targets: ['slurp-rl-tuner:8000'] + metrics_path: '/metrics' \ No newline at end of file