Files

tony fccedf229a Update documentation for website submodule integration

- Add website to project structure documentation
- Include website build commands in management reference
- Document new submodule in architecture overview
- Update .gitmodules with branch specification for better submodule management

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>

2025-08-02 14:02:12 +10:00

13 KiB

Raw Permalink Blame History

🎵 CHORUS Services - Distributed AI Orchestration Platform

CHORUS Services eliminates context loss, reduces hallucinations, and enables scalable multi-agent collaboration through intelligent context management and distributed reasoning.

🚀 Quick Start

# Initialize submodules (first time only)
./chorus.sh init

# Login to Docker registry
./chorus.sh login

# Build and push images to registry
./chorus.sh build

# Start all services
./chorus.sh start

# Check service health
./chorus.sh health

Access Points:

🌐 Marketing Website: https://www.chorus.services (production)
🎛️ Dashboard: https://dashboard.chorus.services (production)
📡 API: https://api.chorus.services (production)
📊 Grafana Monitoring: http://localhost:3002 (admin/chorusadmin)
🔍 Prometheus Metrics: http://localhost:9092

Local Development:

Dashboard: http://localhost:3001
API Docs: http://localhost:8087/docs

🏗️ Architecture Overview

CHORUS Services integrates five core components into a unified platform:

┌─────────────────────────────────────────────────────────────────┐
│                     CHORUS Services Platform                     │
├─────────────────────────────────────────────────────────────────┤
│  🎛️  WHOOSH Orchestrator    │  📊  Monitoring & Analytics      │
│  - Workflow Management       │  - Prometheus Metrics            │
│  - Agent Coordination        │  - Grafana Dashboards            │  
│  - Task Distribution         │  - Real-time Health Monitoring   │
├─────────────────────────────────────────────────────────────────┤
│  🐝 BZZZ P2P Network        │  🧠 SLURP Context Management     │
│  - Agent Mesh Networking    │  - Hierarchical Context Storage  │
│  - Peer Discovery           │  - Semantic Search & Indexing    │
│  - Distributed Coordination │  - Multi-language SDK Support    │
├─────────────────────────────────────────────────────────────────┤
│  🎯 COOEE Feedback System (RL Context SLURP Integration)       │
│  - Performance-based Learning  │  - Context Relevance Tuning   │
│  - Agent Feedback Collection   │  - Role-based Access Control  │
└─────────────────────────────────────────────────────────────────┘

🧩 Core Components

🎛️ WHOOSH - Orchestration Engine

Enterprise workflow management for AI agents
Visual workflow editor with React Flow
Real-time performance monitoring and metrics
Multi-agent task distribution and coordination

🐝 BZZZ - P2P Agent Coordination

Mesh networking with libp2p for resilient communication
Automatic peer discovery via mDNS
Distributed task coordination without single points of failure
Go-based high-performance networking layer

🧠 SLURP - Context Curator Service

Context curation from Hypercore logs based on agent roles and triggers
Role-based context filtering for permissions, deprecation, feature changes
SQL-based context delivery with intelligent relevance scoring
Integration with HCFS for transparent filesystem-based context access

🎯 COOEE - Feedback & Learning (RL Context SLURP)

Reinforcement learning for context relevance tuning
Agent feedback collection with upvote/downvote systems
Role-based context filtering and access control
Continuous improvement through real-world performance data

📊 Monitoring & Analytics

Prometheus metrics collection across all services
Grafana dashboards for visualization and alerting
Health checks and performance monitoring
Audit trails with complete traceability

🛠️ Management Commands

The ./chorus.sh script provides unified management:

# Service Management
./chorus.sh start          # Start all services (registry images)
./chorus.sh stop           # Stop all services  
./chorus.sh restart        # Restart all services
./chorus.sh status         # Show service status
./chorus.sh dev            # Start in development mode (local builds)

# Docker Registry Operations  
./chorus.sh login          # Login to Docker registry
./chorus.sh build          # Build and push all images to registry
./chorus.sh pull           # Pull latest images from registry

# Individual Component Builds
./build-and-push.sh website        # Build and push website only
./build-and-push.sh whoosh-backend # Build and push WHOOSH backend only
./build-and-push.sh bzzz           # Build and push BZZZ coordinator only

# Production Deployment
./chorus.sh deploy         # Deploy to Docker Swarm (production)
./chorus.sh undeploy       # Remove from Docker Swarm

# Development & Maintenance
./chorus.sh update         # Update submodules to latest
./chorus.sh logs [service] # View logs
./chorus.sh health         # Check service health
./chorus.sh clean          # Clean up resources

# First-time Setup
./chorus.sh init           # Initialize git submodules

🌐 Service Endpoints

Service	Port	Purpose	Health Check
WHOOSH Dashboard	3001	Web UI for orchestration	http://localhost:3001
WHOOSH API	8087	REST API + WebSocket	http://localhost:8087/health
BZZZ Coordinator	8080	P2P coordination API	http://localhost:8080/health
SLURP API	8088	Context management API	http://localhost:8088/health
COOEE RL Tuner	8089	Feedback & learning API	http://localhost:8089/health
Grafana	3002	Monitoring dashboards	http://localhost:3002
Prometheus	9092	Metrics collection	http://localhost:9092
PostgreSQL	5433	Database	-
Redis	6380	Cache & message queue	-

📁 Project Structure

chorus.services/
├── 📋 README.md                    # This file
├── 🎛️ chorus.sh                   # Management script
├── 🐳 docker-compose.yml          # Service orchestration
├── 🗄️ init-db.sql                 # Database initialization
├── 📊 monitoring/                  # Monitoring configuration
│   ├── prometheus.yml
│   └── grafana/
├── 📚 docs/                        # Project documentation
│   ├── PROJECT_PLAN.md
│   ├── DESIGN.md
│   └── Copywriting.md
└── 🧩 modules/                     # Git submodules
    ├── whoosh/                     # Orchestration platform
    ├── bzzz/                       # P2P coordination
    ├── slurp/                      # Context management (HCFS)
    └── website/                    # Marketing website (Next.js)

⚙️ Configuration

Environment Variables

Key configuration options in docker-compose.yml:

Database: DATABASE_URL, POSTGRES_*
Redis: REDIS_URL
CORS: CORS_ORIGINS
Logging: LOG_LEVEL
Environment: ENVIRONMENT (development/production)

Adding New Agents

Edit agent configurations in modules/whoosh/config/whoosh.yaml:

whoosh:
  agents:
    my_agent:
      name: "My Agent"
      endpoint: "http://192.168.1.100:11434"
      model: "llama3.1"
      specialization: "coding"
      capabilities: ["python", "javascript"]

🔧 Development Setup

Prerequisites

Docker and Docker Compose
Git
8GB+ RAM recommended
Access to Ollama agents on network

First-Time Setup

# Clone with submodules
git clone --recursive <repository-url>
cd chorus.services

# Or initialize submodules if already cloned
./chorus.sh init

# Login to Docker registry
./chorus.sh login

# Build and push all images to registry
./chorus.sh build

# Start services
./chorus.sh start

Development Workflow

# Development mode (local builds, live reloading)
./chorus.sh dev

# Update submodules to latest
./chorus.sh update

# Rebuild and push after changes
./chorus.sh build

# Pull latest images from registry
./chorus.sh pull

# View logs during development
./chorus.sh logs

# Check service health
./chorus.sh health

# Restart after changes
./chorus.sh restart

Production Deployment

# Deploy to Docker Swarm (production)
./chorus.sh deploy

# Access at https://*.home.deepblack.cloud endpoints

# Remove from swarm
./chorus.sh undeploy

🚀 Git Submodules Guide

CHORUS Services uses git submodules to integrate independent components:

Basic Submodule Commands

# Initialize submodules (first time)
git submodule init
git submodule update

# Update to latest commits
git submodule update --remote

# Check submodule status
git submodule status

# Enter a submodule to work on it
cd modules/whoosh
git checkout main
# Make changes, commit, push

# Return to main project and commit submodule updates
cd ../..
git add modules/whoosh
git commit -m "Update whoosh submodule"

Working with Submodules

Each submodule is an independent git repository
Changes within submodules must be committed in the submodule first
Parent project tracks specific commits of each submodule
Use ./chorus.sh update to pull latest changes from all submodules

📊 Monitoring & Metrics

Key Metrics Tracked

Agent Performance: Response time, throughput, availability
Context Management: Search performance, storage efficiency
P2P Network: Peer connectivity, message latency
System Health: CPU, memory, GPU utilization
Workflow Execution: Success rate, execution time

Grafana Dashboards

CHORUS Overview: Platform-wide health and metrics
Agent Performance: Individual agent monitoring
Context Analytics: SLURP usage and performance
Network Health: BZZZ P2P network monitoring

🛡️ Security Features

Authentication: JWT tokens and API key support
Role-based Access: Context filtering by agent roles
Audit Trails: Complete logging of all operations
Network Security: Internal container networking
Data Privacy: Encrypted context storage

🤝 Contributing

Component Development

Each component can be developed independently:

# Work on WHOOSH orchestrator
cd modules/whoosh
# Follow component-specific development guide

# Work on BZZZ P2P system  
cd modules/bzzz
# Follow Go development practices

# Work on SLURP context system
cd modules/slurp
# Follow Python development practices

Integration Testing

# Test full platform integration
./chorus.sh start
./chorus.sh health

# Run component-specific tests
cd modules/[component]
# Follow component test procedures

📈 Performance Metrics

Production-Ready Performance:

API Response Times: <5ms cached, <50ms uncached
Context Search: <100ms semantic search across 1000+ contexts
P2P Network: <10ms peer communication latency
Workflow Execution: Support for complex multi-agent workflows
Concurrent Agents: Scales to 10+ simultaneous agents

🎯 Use Cases

Enterprise AI Development

Multi-agent software development with specialized AI agents
Context-aware code generation with organizational knowledge
Distributed task execution across development infrastructure

Research & Collaboration

AI agent coordination research with real-world deployment
Context management studies with hierarchical storage
Distributed systems research with P2P networking

Production AI Systems

Enterprise AI orchestration with monitoring and compliance
Context-aware AI applications with persistent memory
Scalable multi-agent deployments with automatic coordination

📞 Support & Documentation

🛠️ Management: ./chorus.sh for all operations
📋 Component Docs: See individual modules/*/README.md
🔧 API Documentation: http://localhost:8087/docs (when running)
📊 Monitoring: http://localhost:3002 (Grafana dashboards)

🎉 Welcome to CHORUS Services!

CHORUS Services represents the future of distributed AI orchestration, providing enterprise-ready tools for context management, agent coordination, and intelligent workflow execution.

🎵 "Individual components make music, but CHORUS Services creates symphony."

Ready to orchestrate your AI agents?

./chorus.sh start

13 KiB Raw Permalink Blame History