diff --git a/docs/comprehensive/README.md b/docs/comprehensive/README.md new file mode 100644 index 0000000..afb4c06 --- /dev/null +++ b/docs/comprehensive/README.md @@ -0,0 +1,226 @@ +# CHORUS Complete Documentation + +**Version:** 1.0.0 +**Generated:** 2025-09-30 +**Status:** Complete comprehensive documentation of CHORUS system + +--- + +## Table of Contents + +### 1. [Architecture Overview](architecture/README.md) +High-level system architecture, design principles, and component relationships + +- [System Architecture](architecture/system-architecture.md) +- [Component Map](architecture/component-map.md) +- [Data Flow](architecture/data-flow.md) +- [Security Architecture](architecture/security.md) +- [Deployment Architecture](architecture/deployment.md) + +### 2. [Command-Line Tools](commands/README.md) +Entry points and command-line interfaces + +- [chorus-agent](commands/chorus-agent.md) - Autonomous agent binary +- [chorus-hap](commands/chorus-hap.md) - Human Agent Portal +- [chorus](commands/chorus.md) - Compatibility wrapper (deprecated) + +### 3. [Core Packages](packages/README.md) +Public API packages in `pkg/` + +#### Execution & AI +- [pkg/execution](packages/execution.md) - Task execution engine and Docker sandboxing +- [pkg/ai](packages/ai.md) - AI provider interfaces and abstractions +- [pkg/providers](packages/providers.md) - Concrete AI provider implementations + +#### Coordination & Distribution +- [pkg/slurp](packages/slurp/README.md) - Distributed coordination system + - [alignment](packages/slurp/alignment.md) - Goal alignment + - [context](packages/slurp/context.md) - Context management + - [distribution](packages/slurp/distribution.md) - Work distribution + - [intelligence](packages/slurp/intelligence.md) - Intelligence layer + - [leader](packages/slurp/leader.md) - Leadership coordination + - [roles](packages/slurp/roles.md) - Role assignments + - [storage](packages/slurp/storage.md) - Distributed storage + - [temporal](packages/slurp/temporal.md) - Time-based coordination +- [pkg/coordination](packages/coordination.md) - Task coordination primitives +- [pkg/election](packages/election.md) - Leader election algorithms +- [pkg/dht](packages/dht.md) - Distributed hash table + +#### Security & Cryptography +- [pkg/crypto](packages/crypto.md) - Encryption and cryptographic primitives +- [pkg/shhh](packages/shhh.md) - Secrets management system +- [pkg/security](packages/security.md) - Security policies and validation + +#### Validation & Compliance +- [pkg/ucxl](packages/ucxl.md) - UCXL validation and enforcement +- [pkg/ucxi](packages/ucxi.md) - UCXI integration + +#### Infrastructure +- [pkg/mcp](packages/mcp.md) - Model Context Protocol implementation +- [pkg/repository](packages/repository.md) - Git repository operations +- [pkg/metrics](packages/metrics.md) - Monitoring and telemetry +- [pkg/health](packages/health.md) - Health check system +- [pkg/config](packages/config.md) - Configuration management +- [pkg/bootstrap](packages/bootstrap.md) - System bootstrapping +- [pkg/pubsub](packages/pubsub.md) - Pub/sub messaging +- [pkg/storage](packages/storage.md) - Storage abstractions +- [pkg/types](packages/types.md) - Common type definitions +- [pkg/version](packages/version.md) - Version information +- [pkg/web](packages/web.md) - Web server and static assets +- [pkg/agentid](packages/agentid.md) - Agent identity management +- [pkg/prompt](packages/prompt.md) - Prompt management +- [pkg/shutdown](packages/shutdown.md) - Graceful shutdown coordination +- [pkg/hmmm](packages/hmmm.md) - HMMM integration +- [pkg/hmmm_adapter](packages/hmmm_adapter.md) - HMMM adapter +- [pkg/integration](packages/integration.md) - Integration utilities +- [pkg/protocol](packages/protocol.md) - Protocol definitions + +### 4. [Internal Packages](internal/README.md) +Private implementation packages in `internal/` + +- [internal/agent](internal/agent.md) - Agent core implementation +- [internal/hapui](internal/hapui.md) - Human Agent Portal UI +- [internal/licensing](internal/licensing.md) - License validation and enforcement +- [internal/logging](internal/logging.md) - Logging infrastructure +- [internal/config](internal/config.md) - Internal configuration +- [internal/runtime](internal/runtime.md) - Runtime environment +- [internal/backbeat](internal/backbeat.md) - Background processing +- [internal/p2p](internal/p2p.md) - Peer-to-peer networking + +### 5. [API Layer](api/README.md) +HTTP API and external interfaces + +- [API Overview](api/overview.md) +- [HTTP Server](api/http-server.md) +- [Setup Manager](api/setup-manager.md) +- [Authentication](api/authentication.md) +- [API Reference](api/reference.md) + +### 6. [Deployment](deployment/README.md) +Deployment configurations and procedures + +- [Docker Setup](deployment/docker.md) +- [Configuration Files](deployment/configuration.md) +- [Environment Variables](deployment/environment.md) +- [Production Deployment](deployment/production.md) +- [Development Setup](deployment/development.md) + +### 7. [Diagrams](diagrams/README.md) +Visual documentation and architecture diagrams + +- [System Overview](diagrams/system-overview.md) +- [Component Interactions](diagrams/component-interactions.md) +- [Sequence Diagrams](diagrams/sequences.md) +- [Data Flow Diagrams](diagrams/data-flow.md) + +--- + +## Quick Reference + +### Key Components + +| Component | Purpose | Status | Location | +|-----------|---------|--------|----------| +| chorus-agent | Autonomous AI agent | Production | cmd/agent | +| Task Execution Engine | Sandboxed code execution | Production | pkg/execution | +| SLURP | Distributed coordination | Production | pkg/slurp | +| UCXL Validation | Compliance enforcement | Production | pkg/ucxl | +| Crypto/SHHH | Security & secrets | Production | pkg/crypto, pkg/shhh | +| HAP | Human Agent Portal | Beta | cmd/hap, internal/hapui | +| MCP Integration | Model Context Protocol | Beta | pkg/mcp | +| DHT | Distributed hash table | Alpha | pkg/dht | +| AI Providers | Multi-provider AI | Production | pkg/ai, pkg/providers | + +### Implementation Status Legend + +- βœ… **Production**: Fully implemented, tested, and production-ready +- πŸ”Ά **Beta**: Implemented with core features, undergoing testing +- πŸ”· **Alpha**: Basic implementation, experimental +- πŸ”΄ **Stubbed**: Interface defined, implementation incomplete +- βšͺ **Mocked**: Mock/simulation for development + +### File Statistics + +- **Total Go files**: 221 (excluding vendor) +- **Packages**: 30+ public packages in `pkg/` +- **Internal packages**: 8 in `internal/` +- **Entry points**: 3 in `cmd/` +- **Lines of code**: ~50,000+ (estimated, excluding vendor) + +--- + +## How to Use This Documentation + +### For New Developers +1. Start with [Architecture Overview](architecture/README.md) +2. Read [System Architecture](architecture/system-architecture.md) +3. Explore [Command-Line Tools](commands/README.md) +4. Deep dive into specific [packages](packages/README.md) as needed + +### For Understanding a Specific Feature +1. Check the [Component Map](architecture/component-map.md) +2. Read the specific package documentation +3. Review relevant [diagrams](diagrams/README.md) +4. See [API Reference](api/reference.md) if applicable + +### For Deployment +1. Read [Deployment Overview](deployment/README.md) +2. Follow [Docker Setup](deployment/docker.md) +3. Configure using [Configuration Files](deployment/configuration.md) +4. Review [Production Deployment](deployment/production.md) + +### For Contributing +1. Understand [Architecture Overview](architecture/README.md) +2. Review relevant package documentation +3. Check implementation status in component tables +4. Follow coding patterns shown in examples + +--- + +## Documentation Conventions + +### Code References +- File paths are shown relative to repository root: `pkg/execution/engine.go` +- Line numbers included when specific: `pkg/execution/engine.go:125-150` +- Functions referenced with parentheses: `ExecuteTask()`, `NewEngine()` +- Types referenced without parentheses: `TaskExecutionRequest`, `Engine` + +### Status Indicators +- **[PRODUCTION]** - Fully implemented and tested +- **[BETA]** - Core features complete, testing in progress +- **[ALPHA]** - Basic implementation, experimental +- **[STUB]** - Interface defined, implementation incomplete +- **[MOCK]** - Simulated/mocked for development +- **[DEPRECATED]** - Scheduled for removal + +### Cross-References +- Internal links use relative paths: [See execution engine](packages/execution.md) +- External links use full URLs: [Docker Documentation](https://docs.docker.com/) +- Code references link to specific sections: [TaskExecutionEngine](packages/execution.md#taskexecutionengine) + +### Diagrams +- ASCII diagrams for simple flows +- Mermaid diagrams for complex relationships (convert to SVG with pandoc) +- Sequence diagrams for interactions +- Component diagrams for architecture + +--- + +## Maintenance + +This documentation was generated through comprehensive code analysis and should be updated when: +- New packages are added +- Significant architectural changes occur +- Implementation status changes (stub β†’ alpha β†’ beta β†’ production) +- APIs change or are deprecated + +To regenerate specific sections, see [Documentation Generation Guide](maintenance.md). + +--- + +## Contact & Support + +For questions about this documentation or the CHORUS system: +- Repository: https://gitea.chorus.services/tony/CHORUS +- Issues: https://gitea.chorus.services/tony/CHORUS/issues +- Documentation issues: Tag with `documentation` label \ No newline at end of file diff --git a/docs/comprehensive/architecture/README.md b/docs/comprehensive/architecture/README.md new file mode 100644 index 0000000..6bfbb11 --- /dev/null +++ b/docs/comprehensive/architecture/README.md @@ -0,0 +1,590 @@ +# CHORUS Architecture Overview + +**System:** CHORUS - Container-First P2P Task Coordination +**Version:** 0.5.0-dev +**Architecture Type:** Distributed, Peer-to-Peer, Event-Driven + +--- + +## Table of Contents + +1. [System Overview](#system-overview) +2. [Core Principles](#core-principles) +3. [Architecture Layers](#architecture-layers) +4. [Key Components](#key-components) +5. [Data Flow](#data-flow) +6. [Deployment Models](#deployment-models) +7. [Related Documents](#related-documents) + +--- + +## System Overview + +CHORUS is a **distributed task coordination system** that enables both autonomous AI agents and human operators to collaborate on software development tasks through a peer-to-peer network. The system provides: + +### Primary Capabilities + +- **Autonomous Agent Execution**: AI agents that can execute code tasks in isolated Docker sandboxes +- **Human-Agent Collaboration**: Human Agent Portal (HAP) for human participation in agent networks +- **Distributed Coordination**: P2P mesh networking with democratic leader election +- **Context Addressing**: UCXL (Universal Context Addressing) for immutable decision tracking +- **Secure Execution**: Multi-layer sandboxing with Docker containers and security policies +- **Collaborative Reasoning**: HMMM protocol for meta-discussion and consensus building +- **Encrypted Storage**: DHT-based encrypted storage for sensitive data + +### System Philosophy + +CHORUS follows these key principles: + +1. **Container-First**: All configuration via environment variables, no file-based config +2. **P2P by Default**: No central server; agents form democratic mesh networks +3. **Zero-Trust Security**: Every operation validated, credentials never stored in containers +4. **Immutable Decisions**: All agent decisions recorded in content-addressed storage +5. **Human-in-the-Loop**: Humans as first-class peers in the agent network + +--- + +## Core Principles + +### 1. Container-Native Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ CHORUS Container β”‚ +β”‚ β”‚ +β”‚ Environment Variables β†’ Runtime Configuration β”‚ +β”‚ Volume Mounts β†’ Prompts & Secrets β”‚ +β”‚ Network Policies β†’ Zero-Egress by Default β”‚ +β”‚ Signal Handling β†’ Dynamic Reconfiguration (SIGHUP) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +**Key Features:** +- No config files inside containers +- All settings via environment variables +- Secrets injected via secure volumes +- Dynamic assignment loading from WHOOSH +- SIGHUP-triggered reconfiguration + +### 2. Peer-to-Peer Mesh Network + +``` + Agent-1 (Alice) + /|\ + / | \ + / | \ + / | \ + Agent-2 | Agent-4 + (Bob) | (Dave) + \ | / + \ | / + \ | / + \|/ + Agent-3 (Carol) + +All agents are equal peers +No central coordinator +Democratic leader election +mDNS local discovery +DHT global discovery +``` + +### 3. Multi-Layer Security + +``` +Layer 1: License Validation (KACHING) + ↓ +Layer 2: P2P Encryption (libp2p TLS) + ↓ +Layer 3: DHT Encryption (age encryption) + ↓ +Layer 4: Docker Sandboxing (namespaces, cgroups) + ↓ +Layer 5: Network Isolation (zero-egress) + ↓ +Layer 6: SHHH Secrets Detection (scan & redact) + ↓ +Layer 7: UCXL Validation (immutable audit trail) + ↓ +Layer 8: Credential Mediation (agent uploads, not container) +``` + +--- + +## Architecture Layers + +CHORUS is organized into distinct architectural layers: + +### Layer 1: P2P Infrastructure + +**Components:** +- libp2p Host (networking) +- mDNS Discovery (local peers) +- DHT (global peer discovery) +- PubSub (message broadcasting) + +**Responsibilities:** +- Peer discovery and connection management +- Encrypted peer-to-peer communication +- Message routing and delivery +- Network resilience and failover + +**See:** [P2P Infrastructure](../internal/p2p.md) + +### Layer 2: Coordination & Consensus + +**Components:** +- Election Manager (leader election) +- Task Coordinator (work distribution) +- HMMM Router (meta-discussion) +- SLURP (distributed orchestration) + +**Responsibilities:** +- Democratic leader election +- Task assignment and tracking +- Collaborative reasoning protocols +- Work distribution algorithms + +**See:** [Coordination](../packages/coordination.md), [SLURP](../packages/slurp/README.md) + +### Layer 3: Execution Engine + +**Components:** +- Task Execution Engine +- Docker Sandbox +- Image Selector +- Command Executor + +**Responsibilities:** +- Isolated code execution in Docker containers +- Language-specific environment selection +- Resource limits and monitoring +- Result capture and validation + +**See:** [Execution Engine](../packages/execution.md), [Task Execution Engine Module](../../Modules/TaskExecutionEngine.md) + +### Layer 4: AI Integration + +**Components:** +- AI Provider Interface +- Provider Implementations (Ollama, ResetData) +- Model Selection Logic +- Prompt Management + +**Responsibilities:** +- Abstract AI provider differences +- Route requests to appropriate models +- Manage system prompts and context +- Handle AI provider failover + +**See:** [AI Providers](../packages/ai.md), [Providers](../packages/providers.md) + +### Layer 5: Storage & State + +**Components:** +- DHT Storage (distributed) +- Encrypted Storage (age encryption) +- UCXL Decision Publisher +- Hypercore Log (append-only) + +**Responsibilities:** +- Distributed data storage +- Encryption and key management +- Immutable decision recording +- Event log persistence + +**See:** [DHT](../packages/dht.md), [UCXL](../packages/ucxl.md) + +### Layer 6: Security & Validation + +**Components:** +- License Validator (KACHING) +- SHHH Sentinel (secrets detection) +- Crypto Layer (encryption) +- Security Policies + +**Responsibilities:** +- License enforcement +- Secrets scanning and redaction +- Cryptographic operations +- Security policy enforcement + +**See:** [Crypto](../packages/crypto.md), [SHHH](../packages/shhh.md), [Licensing](../internal/licensing.md) + +### Layer 7: Observability + +**Components:** +- Metrics Collector (CHORUS Metrics) +- Health Checks (liveness, readiness) +- BACKBEAT Integration (P2P telemetry) +- Hypercore Log (coordination events) + +**Responsibilities:** +- System metrics collection +- Health monitoring +- P2P operation tracking +- Event logging and audit trails + +**See:** [Metrics](../packages/metrics.md), [Health](../packages/health.md) + +### Layer 8: External Interfaces + +**Components:** +- HTTP API Server +- UCXI Server (content resolution) +- HAP Terminal Interface +- HAP Web Interface [STUB] + +**Responsibilities:** +- REST API endpoints +- UCXL content resolution +- Human interaction interfaces +- External system integration + +**See:** [API](../api/README.md), [UCXI](../packages/ucxi.md), [HAP UI](../internal/hapui.md) + +--- + +## Key Components + +### Runtime Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ main.go (cmd/agent or cmd/hap) β”‚ +β”‚ β”‚ β”‚ +β”‚ └─→ internal/runtime.Initialize() β”‚ +β”‚ β”‚ β”‚ +β”‚ β”œβ”€β†’ Config Loading (environment) β”‚ +β”‚ β”œβ”€β†’ License Validation (KACHING) β”‚ +β”‚ β”œβ”€β†’ AI Provider Setup (Ollama/ResetData) β”‚ +β”‚ β”œβ”€β†’ P2P Node Creation (libp2p) β”‚ +β”‚ β”œβ”€β†’ PubSub Initialization β”‚ +β”‚ β”œβ”€β†’ DHT Setup (optional) β”‚ +β”‚ β”œβ”€β†’ Election Manager β”‚ +β”‚ β”œβ”€β†’ Task Coordinator β”‚ +β”‚ β”œβ”€β†’ HTTP API Server β”‚ +β”‚ β”œβ”€β†’ UCXI Server (optional) β”‚ +β”‚ └─→ Health & Metrics β”‚ +β”‚ β”‚ +β”‚ SharedRuntime β”‚ +β”‚ β”œβ”€β”€ Context & Cancellation β”‚ +β”‚ β”œβ”€β”€ Logger (SimpleLogger) β”‚ +β”‚ β”œβ”€β”€ Config (*config.Config) β”‚ +β”‚ β”œβ”€β”€ RuntimeConfig (dynamic assignments) β”‚ +β”‚ β”œβ”€β”€ P2P Node (*p2p.Node) β”‚ +β”‚ β”œβ”€β”€ PubSub (*pubsub.PubSub) β”‚ +β”‚ β”œβ”€β”€ DHT (*dht.LibP2PDHT) β”‚ +β”‚ β”œβ”€β”€ Encrypted Storage (*dht.EncryptedDHTStorage) β”‚ +β”‚ β”œβ”€β”€ Election Manager (*election.ElectionManager) β”‚ +β”‚ β”œβ”€β”€ Task Coordinator (*coordinator.TaskCoordinator) β”‚ +β”‚ β”œβ”€β”€ HTTP Server (*api.HTTPServer) β”‚ +β”‚ β”œβ”€β”€ UCXI Server (*ucxi.Server) β”‚ +β”‚ β”œβ”€β”€ Health Manager (*health.Manager) β”‚ +β”‚ β”œβ”€β”€ Metrics (*metrics.CHORUSMetrics) β”‚ +β”‚ β”œβ”€β”€ SHHH Sentinel (*shhh.Sentinel) β”‚ +β”‚ β”œβ”€β”€ BACKBEAT Integration (*backbeat.Integration) β”‚ +β”‚ └── Decision Publisher (*ucxl.DecisionPublisher) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### Binary Separation + +CHORUS provides three binaries with shared infrastructure: + +| Binary | Purpose | Mode | Status | +|--------|---------|------|--------| +| **chorus-agent** | Autonomous AI agent | Agent Mode | βœ… Production | +| **chorus-hap** | Human Agent Portal | HAP Mode | πŸ”Ά Beta | +| **chorus** | Compatibility wrapper | N/A | πŸ”΄ Deprecated | + +All binaries share: +- P2P infrastructure (libp2p, PubSub, DHT) +- Election and coordination systems +- Security and encryption layers +- Configuration and licensing + +Differences: +- **Agent**: Automatic task execution, autonomous reasoning +- **HAP**: Terminal/web UI for human interaction, manual task approval + +**See:** [Commands](../commands/README.md) + +--- + +## Data Flow + +### Task Execution Flow + +``` +1. Task Request Arrives + β”‚ + β”œβ”€β†’ Via PubSub (from another agent) + β”œβ”€β†’ Via HTTP API (from external system) + └─→ Via HAP (from human operator) + β”‚ + ↓ +2. Task Coordinator Receives Task + β”‚ + β”œβ”€β†’ Check agent availability + β”œβ”€β†’ Validate task structure + └─→ Assign to execution engine + β”‚ + ↓ +3. Execution Engine Processes + β”‚ + β”œβ”€β†’ Detect language (Go, Rust, Python, etc.) + β”œβ”€β†’ Select Docker image + β”œβ”€β†’ Create sandbox configuration + β”œβ”€β†’ Start container + β”‚ β”‚ + β”‚ β”œβ”€β†’ Mount /workspace/input (read-only source) + β”‚ β”œβ”€β†’ Mount /workspace/data (working directory) + β”‚ └─→ Mount /workspace/output (deliverables) + β”‚ + β”œβ”€β†’ Execute commands via Docker Exec API + β”œβ”€β†’ Stream stdout/stderr + β”œβ”€β†’ Monitor resource usage + └─→ Capture exit codes + β”‚ + ↓ +4. Result Processing + β”‚ + β”œβ”€β†’ Collect artifacts from /workspace/output + β”œβ”€β†’ Generate task summary + β”œβ”€β†’ Create UCXL decision record + └─→ Publish to DHT (encrypted) + β”‚ + ↓ +5. Result Distribution + β”‚ + β”œβ”€β†’ Broadcast completion via PubSub + β”œβ”€β†’ Update task tracker (availability) + β”œβ”€β†’ Notify requester (if HTTP API) + └─→ Log to Hypercore (audit trail) +``` + +### Decision Publishing Flow + +``` +Agent Decision Made + β”‚ + ↓ +Generate UCXL Context Address + β”‚ + β”œβ”€β†’ Hash decision content (SHA-256) + β”œβ”€β†’ Create ucxl:// URI + └─→ Add metadata (agent ID, timestamp) + β”‚ + ↓ +Encrypt Decision Data + β”‚ + β”œβ”€β†’ Use age encryption + β”œβ”€β†’ Derive key from shared secret + └─→ Create encrypted blob + β”‚ + ↓ +Store in DHT + β”‚ + β”œβ”€β†’ Key: UCXL hash + β”œβ”€β†’ Value: Encrypted decision + └─→ TTL: Configured expiration + β”‚ + ↓ +Announce on PubSub + β”‚ + β”œβ”€β†’ Topic: "chorus/decisions" + β”œβ”€β†’ Payload: UCXL address only + └─→ Interested peers can fetch from DHT +``` + +### Election Flow + +``` +Agent Startup + β”‚ + ↓ +Join Election Topic + β”‚ + β”œβ”€β†’ Subscribe to "chorus/election/v1" + β”œβ”€β†’ Announce presence + └─→ Share capabilities + β”‚ + ↓ +Send Heartbeats + β”‚ + β”œβ”€β†’ Every 5 seconds + β”œβ”€β†’ Include: Node ID, Uptime, Load + └─→ Track other peers' heartbeats + β”‚ + ↓ +Monitor Admin Status + β”‚ + β”œβ”€β†’ Track last admin heartbeat + β”œβ”€β†’ Timeout: 15 seconds + └─→ If timeout β†’ Trigger election + β”‚ + ↓ +Election Triggered + β”‚ + β”œβ”€β†’ All agents propose themselves + β”œβ”€β†’ Vote for highest uptime + β”œβ”€β†’ Consensus on winner + └─→ Winner becomes admin + β”‚ + ↓ +Admin Elected + β”‚ + β”œβ”€β†’ Winner assumes admin role + β”œβ”€β†’ Applies admin configuration + β”œβ”€β†’ Enables SLURP coordination + └─→ Continues heartbeat at higher frequency +``` + +--- + +## Deployment Models + +### Model 1: Local Development + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Developer Laptop β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ chorus-agent β”‚ β”‚ chorus-hap β”‚ β”‚ +β”‚ β”‚ (Alice) β”‚ β”‚ (Human) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ mDNS Discovery β”‚ +β”‚ P2P Mesh (local) β”‚ +β”‚ β”‚ +β”‚ Ollama: localhost:11434 β”‚ +β”‚ Docker: /var/run/docker.sock β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +**Characteristics:** +- Single machine deployment +- mDNS for peer discovery +- Local Ollama instance +- Shared Docker socket +- No DHT required + +**Use Cases:** +- Local testing +- Development workflows +- Single-user tasks + +### Model 2: Docker Swarm Cluster + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Docker Swarm Cluster β”‚ +β”‚ β”‚ +β”‚ Manager Node 1 Manager Node 2 Worker 1 β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ chorus-agent │←─────→│ chorus-agent │←─────→│ chorus β”‚ β”‚ +β”‚ β”‚ (Leader) β”‚ β”‚ (Follower) β”‚ β”‚ -agent β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ ↑ ↑ ↑ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ Docker Swarm Overlay Network β”‚ +β”‚ P2P Mesh + DHT β”‚ +β”‚ β”‚ +β”‚ Shared Services: β”‚ +β”‚ - Docker Registry (private) β”‚ +β”‚ - Ollama Distributed (5 nodes) β”‚ +β”‚ - NFS Storage (/rust) β”‚ +β”‚ - WHOOSH (assignment server) β”‚ +β”‚ - KACHING (license server) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +**Characteristics:** +- Multi-node cluster +- DHT for global discovery +- Bootstrap peers for network joining +- Overlay networking +- Shared storage via NFS +- Centralized license validation + +**Use Cases:** +- Production deployments +- Team collaboration +- High availability +- Scalable workloads + +### Model 3: Hybrid (Agent + HAP) + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Production Environment β”‚ +β”‚ β”‚ +β”‚ Docker Swarm Developer Workstation β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ chorus-agent β”‚ β”‚ chorus-hap β”‚ β”‚ +β”‚ β”‚ (Alice) │←─────P2P─────→│ (Human-Bob) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ chorus-agent β”‚ β”‚ +β”‚ β”‚ (Carol) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β”‚ Autonomous agents run in swarm β”‚ +β”‚ Human operator joins via HAP (local or remote) β”‚ +β”‚ Same P2P protocol, equal participants β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +**Characteristics:** +- Autonomous agents in production +- Human operators join as needed +- Collaborative decision-making +- HMMM meta-discussion +- Humans can override or guide + +**Use Cases:** +- Supervised automation +- Human-in-the-loop workflows +- Critical decision points +- Training and oversight + +--- + +## Related Documents + +### Getting Started +- [Commands Overview](../commands/README.md) - Entry points and CLI tools +- [Deployment Guide](../deployment/README.md) - How to deploy CHORUS +- [Configuration](../deployment/configuration.md) - Environment variables and settings + +### Core Systems +- [Task Execution Engine](../../Modules/TaskExecutionEngine.md) - Complete execution engine documentation +- [P2P Infrastructure](../internal/p2p.md) - libp2p networking details +- [SLURP System](../packages/slurp/README.md) - Distributed coordination + +### Security +- [Security Architecture](security.md) - Security layers and threat model +- [Crypto Package](../packages/crypto.md) - Encryption and key management +- [SHHH](../packages/shhh.md) - Secrets detection and redaction +- [Licensing](../internal/licensing.md) - License validation + +### Integration +- [API Reference](../api/reference.md) - HTTP API endpoints +- [UCXL System](../packages/ucxl.md) - Context addressing +- [AI Providers](../packages/ai.md) - AI integration + +--- + +## Next Steps + +For detailed information on specific components: +1. **New to CHORUS?** Start with [System Architecture](system-architecture.md) +2. **Want to deploy?** See [Deployment Guide](../deployment/README.md) +3. **Developing features?** Review [Component Map](component-map.md) +4. **Understanding execution?** Read [Task Execution Engine](../../Modules/TaskExecutionEngine.md) \ No newline at end of file diff --git a/docs/comprehensive/commands/chorus-agent.md b/docs/comprehensive/commands/chorus-agent.md new file mode 100644 index 0000000..17c1025 --- /dev/null +++ b/docs/comprehensive/commands/chorus-agent.md @@ -0,0 +1,738 @@ +# chorus-agent - Autonomous Agent Binary + +**Binary:** `chorus-agent` +**Source:** `cmd/agent/main.go` +**Status:** βœ… Production +**Purpose:** Autonomous AI agent for P2P task coordination + +--- + +## Overview + +`chorus-agent` is the primary executable for running autonomous AI agents in the CHORUS system. Agents participate in peer-to-peer networks, execute tasks in isolated Docker sandboxes, collaborate with other agents via HMMM protocol, and maintain distributed state through DHT storage. + +### Key Features + +- βœ… **Autonomous Operation**: Executes tasks without human intervention +- βœ… **P2P Networking**: Participates in distributed mesh network +- βœ… **Docker Sandboxing**: Isolated code execution environments +- βœ… **HMMM Reasoning**: Collaborative meta-discussion protocol +- βœ… **DHT Storage**: Encrypted distributed data storage +- βœ… **UCXL Publishing**: Immutable decision recording +- βœ… **Democratic Elections**: Participates in leader election +- βœ… **Health Monitoring**: Self-reporting health status + +--- + +## Usage + +### Basic Invocation + +```bash +# With required environment variables +CHORUS_LICENSE_ID=dev-123 \ +CHORUS_AGENT_ID=chorus-agent-1 \ +./chorus-agent +``` + +### Help Output + +```bash +$ ./chorus-agent --help +CHORUS-agent 0.5.0-dev (build: abc123, 2025-09-30) + +Usage: + chorus-agent [--help] [--version] + +CHORUS Autonomous Agent - P2P Task Coordination + +This binary runs autonomous AI agents that participate in P2P task coordination, +collaborative reasoning via HMMM, and distributed decision making. + +Environment (common): + CHORUS_LICENSE_ID (required) + CHORUS_AGENT_ID (optional; auto-generated if empty) + CHORUS_P2P_PORT (default 9000) + CHORUS_API_PORT (default 8080) + CHORUS_HEALTH_PORT (default 8081) + CHORUS_DHT_ENABLED (default true) + CHORUS_BOOTSTRAP_PEERS (comma-separated multiaddrs) + OLLAMA_ENDPOINT (default http://localhost:11434) + +Example: + CHORUS_LICENSE_ID=dev-123 \ + CHORUS_AGENT_ID=chorus-agent-1 \ + CHORUS_P2P_PORT=9000 CHORUS_API_PORT=8080 ./chorus-agent + +Agent Features: + - Autonomous task execution + - P2P mesh networking + - HMMM collaborative reasoning + - DHT encrypted storage + - UCXL context addressing + - Democratic leader election + - Health monitoring +``` + +### Version Information + +```bash +$ ./chorus-agent --version +CHORUS-agent 0.5.0-dev (build: abc123, 2025-09-30) +``` + +--- + +## Source Code Analysis + +### File: `cmd/agent/main.go` + +**Lines:** 79 +**Package:** main +**Imports:** +- `chorus/internal/runtime` - Shared P2P runtime infrastructure + +### Build-Time Variables + +```go +// Lines 11-16 +var ( + version = "0.5.0-dev" + commitHash = "unknown" + buildDate = "unknown" +) +``` + +**Set via ldflags:** +```bash +go build -ldflags "-X main.version=1.0.0 -X main.commitHash=$(git rev-parse --short HEAD) -X main.buildDate=$(date -u +%Y-%m-%d)" +``` + +### main() Function Flow + +```go +func main() { + // 1. CLI Argument Handling (lines 19-59) + // - Check for --help, -h, help + // - Check for --version, -v + // - Print usage and exit early if found + + // 2. Set Build Information (lines 61-64) + runtime.AppVersion = version + runtime.AppCommitHash = commitHash + runtime.AppBuildDate = buildDate + + // 3. Initialize Shared Runtime (lines 66-72) + sharedRuntime, err := runtime.Initialize("agent") + if err != nil { + // Fatal error, exit 1 + } + defer sharedRuntime.Cleanup() + + // 4. Start Agent Mode (lines 74-78) + if err := sharedRuntime.StartAgentMode(); err != nil { + // Fatal error, exit 1 + } +} +``` + +### Execution Phases + +#### Phase 1: Early CLI Handling (lines 19-59) + +**Purpose:** Handle help/version requests without loading configuration + +**Code:** +```go +for _, a := range os.Args[1:] { + switch a { + case "--help", "-h", "help": + // Print detailed help message + fmt.Printf("%s-agent %s (build: %s, %s)\n\n", runtime.AppName, version, commitHash, buildDate) + // ... usage information ... + return + case "--version", "-v": + fmt.Printf("%s-agent %s (build: %s, %s)\n", runtime.AppName, version, commitHash, buildDate) + return + } +} +``` + +**Why Important:** Allows users to get help without needing valid license or configuration. + +#### Phase 2: Runtime Initialization (line 67) + +**Function Call:** `runtime.Initialize("agent")` + +**What Happens:** +1. Load configuration from environment variables +2. Validate CHORUS license with KACHING server +3. Initialize AI provider (Ollama or ResetData) +4. Create P2P libp2p node +5. Start mDNS discovery +6. Initialize PubSub messaging +7. Setup DHT (if enabled) +8. Start election manager +9. Create task coordinator +10. Start HTTP API server +11. Start UCXI server (if enabled) +12. Initialize health checks +13. Setup SHHH sentinel (secrets detection) +14. Configure metrics collection + +**Returns:** `*runtime.SharedRuntime` containing all initialized components + +**See:** [internal/runtime Documentation](../internal/runtime.md) for complete initialization details + +#### Phase 3: Agent Mode Activation (line 75) + +**Function Call:** `sharedRuntime.StartAgentMode()` + +**What Happens:** +1. Agent registers itself as available for tasks +2. Begins listening for task assignments via PubSub +3. Starts autonomous task execution loops +4. Enables automatic decision making +5. Activates HMMM meta-discussion participation +6. Begins heartbeat broadcasting for election + +**Implementation:** See `internal/runtime/agent_support.go` + +**Behavior Differences from HAP:** +- **Agent**: Automatically accepts and executes tasks +- **HAP**: Prompts human for task approval + +--- + +## Configuration + +### Required Environment Variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `CHORUS_LICENSE_ID` | License key from KACHING | `dev-123` | + +### Optional Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `CHORUS_AGENT_ID` | Auto-generated | Unique agent identifier | +| `CHORUS_P2P_PORT` | 9000 | libp2p listening port | +| `CHORUS_API_PORT` | 8080 | HTTP API port | +| `CHORUS_HEALTH_PORT` | 8081 | Health check port | +| `CHORUS_DHT_ENABLED` | true | Enable distributed hash table | +| `CHORUS_BOOTSTRAP_PEERS` | "" | Comma-separated multiaddrs | +| `OLLAMA_ENDPOINT` | http://localhost:11434 | Ollama API endpoint | + +### Role-Based Configuration + +| Variable | Default | Description | +|----------|---------|-------------| +| `CHORUS_AGENT_ROLE` | "" | Agent role (admin, developer, reviewer) | +| `CHORUS_AGENT_EXPERTISE` | "" | Comma-separated expertise areas | +| `CHORUS_AGENT_REPORTS_TO` | "" | Supervisor agent ID | +| `CHORUS_AGENT_SPECIALIZATION` | "general" | Task specialization | +| `CHORUS_AGENT_MAX_TASKS` | 3 | Max concurrent tasks | + +### AI Provider Configuration + +#### Ollama (Default) + +```bash +export CHORUS_AI_PROVIDER=ollama +export OLLAMA_ENDPOINT=http://192.168.1.72:11434 +``` + +#### ResetData + +```bash +export CHORUS_AI_PROVIDER=resetdata +export RESETDATA_API_KEY=your-api-key-here +export RESETDATA_BASE_URL=https://api.resetdata.ai +export RESETDATA_MODEL=claude-3-5-sonnet-20250930 +``` + +### Assignment Loading + +Agents can load dynamic configuration from WHOOSH: + +```bash +export ASSIGN_URL=https://whoosh.example.com/api/assignments/agent-123.json +``` + +When configured, agents: +1. Fetch assignment JSON on startup +2. Merge with environment config +3. Listen for SIGHUP to reload +4. Update configuration without restart + +**See:** [Configuration Management](../packages/config.md) for assignment schema + +--- + +## Runtime Behavior + +### Startup Sequence + +``` +1. Parse CLI arguments + β”œβ”€β†’ --help β†’ print help, exit 0 + β”œβ”€β†’ --version β†’ print version, exit 0 + └─→ (none) β†’ continue + +2. Set build information in runtime package + +3. Initialize shared runtime + β”œβ”€β†’ Load environment configuration + β”œβ”€β†’ Validate license with KACHING + β”‚ └─→ FAIL β†’ print error, exit 1 + β”œβ”€β†’ Configure AI provider + β”œβ”€β†’ Create P2P node + β”œβ”€β†’ Start mDNS discovery + β”œβ”€β†’ Initialize PubSub + β”œβ”€β†’ Setup DHT (optional) + β”œβ”€β†’ Start election manager + β”œβ”€β†’ Create task coordinator + β”œβ”€β†’ Start HTTP API server + └─→ Initialize health checks + +4. Start agent mode + β”œβ”€β†’ Register as available agent + β”œβ”€β†’ Join task coordination topics + β”œβ”€β†’ Begin heartbeat broadcasting + β”œβ”€β†’ Enable autonomous task execution + └─→ Activate HMMM participation + +5. Run until signal (SIGINT, SIGTERM) + +6. Cleanup on shutdown + β”œβ”€β†’ Stop accepting new tasks + β”œβ”€β†’ Complete in-flight tasks + β”œβ”€β†’ Close P2P connections + β”œβ”€β†’ Flush DHT cache + β”œβ”€β†’ Stop HTTP servers + └─→ Exit gracefully +``` + +### Signal Handling + +| Signal | Behavior | +|--------|----------| +| SIGINT | Graceful shutdown (complete current tasks) | +| SIGTERM | Graceful shutdown (complete current tasks) | +| SIGHUP | Reload configuration from ASSIGN_URL | + +### Task Execution Loop + +Once in agent mode: + +``` +Loop Forever: +β”‚ +β”œβ”€β†’ Listen for tasks on PubSub topic "chorus/tasks" +β”‚ +β”œβ”€β†’ Task received: +β”‚ β”œβ”€β†’ Check agent availability (< max tasks) +β”‚ β”œβ”€β†’ Check task matches specialization +β”‚ └─→ Accept or decline +β”‚ +β”œβ”€β†’ Task accepted: +β”‚ β”œβ”€β†’ Increment active task count +β”‚ β”œβ”€β†’ Log task start to Hypercore +β”‚ β”œβ”€β†’ Invoke execution engine +β”‚ β”‚ β”œβ”€β†’ Select Docker image based on language +β”‚ β”‚ β”œβ”€β†’ Create sandbox container +β”‚ β”‚ β”œβ”€β†’ Execute commands via Docker Exec API +β”‚ β”‚ β”œβ”€β†’ Stream output +β”‚ β”‚ β”œβ”€β†’ Monitor resource usage +β”‚ β”‚ └─→ Capture results +β”‚ β”œβ”€β†’ Generate task summary +β”‚ β”œβ”€β†’ Create UCXL decision record +β”‚ β”œβ”€β†’ Publish decision to DHT +β”‚ β”œβ”€β†’ Broadcast completion on PubSub +β”‚ β”œβ”€β†’ Decrement active task count +β”‚ └─→ Log task completion to Hypercore +β”‚ +└─→ Continue listening +``` + +**See:** [Task Execution Engine](../packages/execution.md) for execution details + +--- + +## P2P Networking + +### Peer Discovery + +**mDNS (Local):** +- Discovers peers on local network +- Service name: `chorus-peer-discovery` +- No configuration required +- Automatic peer connection + +**DHT (Global):** +- Discovers peers across networks +- Requires bootstrap peers +- Content-addressed routing +- Kademlia-based DHT + +**Bootstrap Peers:** +```bash +export CHORUS_BOOTSTRAP_PEERS="/ip4/192.168.1.100/tcp/9000/p2p/12D3KooWABC...,/ip4/192.168.1.101/tcp/9000/p2p/12D3KooWXYZ..." +``` + +### Topics Subscribed + +| Topic | Purpose | +|-------|---------| +| `chorus/coordination/v1` | Task coordination messages | +| `hmmm/meta-discussion/v1` | Collaborative reasoning | +| `chorus/election/v1` | Leader election heartbeats | +| `chorus/decisions` | Decision announcements | +| `chorus/health` | Health status broadcasts | + +### Role-Based Topics (Optional) + +If `CHORUS_AGENT_ROLE` is set, agent also joins: + +| Topic | Purpose | +|-------|---------| +| `chorus/role/{role}` | Role-specific coordination | +| `chorus/expertise/{expertise}` | Expertise-based routing | +| `chorus/reports/{supervisor}` | Reporting hierarchy | + +--- + +## Health Checks + +### HTTP Endpoints + +**Liveness Probe:** +```bash +curl http://localhost:8081/healthz +# Returns: 200 OK if agent is alive +``` + +**Readiness Probe:** +```bash +curl http://localhost:8081/ready +# Returns: 200 OK if agent is ready for tasks +# Returns: 503 Service Unavailable if at max capacity +``` + +**Health Details:** +```bash +curl http://localhost:8081/health +# Returns JSON with: +# - P2P connectivity status +# - DHT reachability +# - Active task count +# - Available capacity +# - Last heartbeat time +``` + +### Health Criteria + +Agent is **healthy** when: +- βœ… License valid +- βœ… P2P node connected +- βœ… At least 1 peer discovered +- βœ… Election manager running +- βœ… Task coordinator active +- βœ… HTTP API responding + +Agent is **ready** when: +- βœ… All health checks pass +- βœ… Active tasks < max tasks +- βœ… Docker daemon reachable +- βœ… AI provider accessible + +**See:** [Health Package](../packages/health.md) + +--- + +## Monitoring & Metrics + +### Prometheus Metrics + +Exposed on `http://localhost:8080/metrics`: + +**Task Metrics:** +- `chorus_tasks_active` - Current active tasks +- `chorus_tasks_completed_total` - Total completed tasks +- `chorus_tasks_failed_total` - Total failed tasks +- `chorus_task_duration_seconds` - Task execution duration histogram + +**P2P Metrics:** +- `chorus_peers_connected` - Number of connected peers +- `chorus_pubsub_messages_sent_total` - PubSub messages sent +- `chorus_pubsub_messages_received_total` - PubSub messages received +- `chorus_dht_queries_total` - DHT query count +- `chorus_dht_cache_hits_total` - DHT cache hits +- `chorus_dht_cache_misses_total` - DHT cache misses + +**Execution Metrics:** +- `chorus_sandbox_containers_active` - Active Docker containers +- `chorus_sandbox_cpu_usage` - Container CPU usage +- `chorus_sandbox_memory_usage_bytes` - Container memory usage + +**Security Metrics:** +- `chorus_shhh_findings_total` - Secrets detected by SHHH +- `chorus_license_checks_total` - License validation attempts +- `chorus_license_failures_total` - Failed license validations + +**See:** [Metrics Package](../packages/metrics.md) + +--- + +## Integration Points + +### WHOOSH Assignment System + +Agents can load dynamic assignments from WHOOSH: + +```bash +# Set assignment URL +export ASSIGN_URL=https://whoosh.example.com/api/assignments/agent-123.json + +# Agent fetches assignment on startup +# Assignment JSON structure: +{ + "agent_id": "agent-123", + "role": "developer", + "expertise": ["rust", "go"], + "reports_to": "agent-admin", + "max_tasks": 5, + "bootstrap_peers": [ + "/ip4/192.168.1.100/tcp/9000/p2p/12D3KooWABC..." + ], + "join_stagger_ms": 5000 +} + +# Reload with SIGHUP +kill -HUP $(pidof chorus-agent) +``` + +### KACHING License Server + +All agents validate licenses on startup: + +```bash +# License validation flow +1. Agent starts with CHORUS_LICENSE_ID +2. Connects to KACHING server (from config) +3. Validates license is: + - Valid and not expired + - Assigned to correct cluster + - Has required permissions +4. If invalid: agent exits with error +5. If valid: agent continues startup +``` + +**See:** [Licensing](../internal/licensing.md) + +### BACKBEAT Integration + +Optional telemetry system for P2P operations: + +```bash +export CHORUS_BACKBEAT_ENABLED=true +export CHORUS_BACKBEAT_ENDPOINT=http://backbeat.example.com + +# When enabled, agent tracks: +# - P2P operation phases +# - DHT bootstrap timing +# - Election progression +# - Task execution phases +``` + +**See:** [BACKBEAT Integration](../internal/backbeat.md) + +--- + +## Example Deployments + +### Local Development + +```bash +#!/bin/bash +# Run local agent for development + +export CHORUS_LICENSE_ID=dev-local-123 +export CHORUS_AGENT_ID=dev-agent-1 +export CHORUS_P2P_PORT=9000 +export CHORUS_API_PORT=8080 +export CHORUS_HEALTH_PORT=8081 +export OLLAMA_ENDPOINT=http://localhost:11434 +export CHORUS_DHT_ENABLED=false # Disable DHT for local dev + +./chorus-agent +``` + +### Docker Container + +```dockerfile +FROM debian:bookworm-slim + +# Install runtime dependencies +RUN apt-get update && apt-get install -y \ + ca-certificates \ + docker.io \ + && rm -rf /var/lib/apt/lists/* + +# Copy binary +COPY chorus-agent /usr/local/bin/chorus-agent + +# Expose ports +EXPOSE 9000 8080 8081 + +# Run as non-root +USER nobody + +ENTRYPOINT ["/usr/local/bin/chorus-agent"] +``` + +```bash +docker run -d \ + --name chorus-agent-1 \ + -e CHORUS_LICENSE_ID=prod-123 \ + -e CHORUS_AGENT_ID=agent-1 \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -p 9000:9000 \ + -p 8080:8080 \ + -p 8081:8081 \ + chorus-agent:latest +``` + +### Docker Swarm Service + +```yaml +version: "3.8" +services: + chorus-agent: + image: registry.example.com/chorus-agent:1.0.0 + environment: + CHORUS_LICENSE_ID: ${CHORUS_LICENSE_ID} + CHORUS_P2P_PORT: 9000 + CHORUS_API_PORT: 8080 + CHORUS_DHT_ENABLED: "true" + CHORUS_BOOTSTRAP_PEERS: "/ip4/192.168.1.100/tcp/9000/p2p/12D3KooWABC..." + ASSIGN_URL: "https://whoosh.example.com/api/assignments/{{.Service.Name}}.{{.Task.Slot}}.json" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - /rust/containers/WHOOSH/prompts:/prompts:ro + deploy: + replicas: 3 + placement: + constraints: + - node.role == worker + networks: + - chorus-mesh + ports: + - target: 9000 + published: 9000 + mode: host +``` + +--- + +## Troubleshooting + +### Agent Won't Start + +**Symptom:** Agent exits immediately with error + +**Possible Causes:** +1. Invalid or missing license + ``` + ❌ Failed to initialize CHORUS agent: license validation failed + ``` + **Fix:** Check `CHORUS_LICENSE_ID` and KACHING server connectivity + +2. Docker socket not accessible + ``` + ❌ Failed to create P2P node: failed to create Docker client + ``` + **Fix:** Mount `/var/run/docker.sock` or check Docker daemon + +3. Port already in use + ``` + ❌ Failed to initialize: bind: address already in use + ``` + **Fix:** Change `CHORUS_P2P_PORT` or kill process on port + +### No Peer Discovery + +**Symptom:** Agent starts but shows 0 connected peers + +**Possible Causes:** +1. mDNS blocked by firewall + **Fix:** Allow UDP port 5353, or use bootstrap peers + +2. No bootstrap peers configured + **Fix:** Set `CHORUS_BOOTSTRAP_PEERS` with valid multiaddrs + +3. Network isolation + **Fix:** Ensure agents can reach each other on P2P ports + +### Tasks Not Executing + +**Symptom:** Agent receives tasks but doesn't execute + +**Possible Causes:** +1. Agent at max capacity + **Check:** `curl localhost:8080/metrics | grep chorus_tasks_active` + **Fix:** Increase `CHORUS_AGENT_MAX_TASKS` + +2. Docker images not available + **Check:** `docker images | grep chorus` + **Fix:** Pull images: `docker pull anthonyrawlins/chorus-rust-dev:latest` + +3. Wrong specialization + **Check:** Task language doesn't match agent expertise + **Fix:** Adjust `CHORUS_AGENT_EXPERTISE` or remove specialization + +### High Memory Usage + +**Symptom:** Agent consuming excessive memory + +**Possible Causes:** +1. DHT cache size too large + **Fix:** Reduce `CHORUS_DHT_CACHE_SIZE` (default 100MB) + +2. Too many concurrent tasks + **Fix:** Reduce `CHORUS_AGENT_MAX_TASKS` + +3. Memory leak in long-running containers + **Fix:** Restart agent periodically or investigate task code + +--- + +## Related Documentation + +- [chorus-hap](chorus-hap.md) - Human Agent Portal binary +- [chorus](chorus.md) - Deprecated compatibility wrapper +- [internal/runtime](../internal/runtime.md) - Shared runtime initialization +- [Task Execution Engine](../packages/execution.md) - Task execution details +- [Configuration](../deployment/configuration.md) - Environment variables reference +- [Deployment](../deployment/docker.md) - Docker deployment guide + +--- + +## Implementation Status + +| Feature | Status | Notes | +|---------|--------|-------| +| P2P Networking | βœ… Production | libp2p, mDNS, DHT | +| Task Execution | βœ… Production | Docker sandboxing | +| License Validation | βœ… Production | KACHING integration | +| HMMM Reasoning | πŸ”Ά Beta | Collaborative meta-discussion | +| UCXL Publishing | βœ… Production | Decision recording | +| Election | βœ… Production | Democratic leader election | +| Health Checks | βœ… Production | Liveness & readiness | +| Metrics | βœ… Production | Prometheus format | +| Assignment Loading | βœ… Production | WHOOSH integration | +| SIGHUP Reload | βœ… Production | Dynamic reconfiguration | +| BACKBEAT Telemetry | πŸ”Ά Beta | Optional P2P tracking | + +**Last Updated:** 2025-09-30 \ No newline at end of file diff --git a/docs/comprehensive/commands/chorus-hap.md b/docs/comprehensive/commands/chorus-hap.md new file mode 100644 index 0000000..02a49b2 --- /dev/null +++ b/docs/comprehensive/commands/chorus-hap.md @@ -0,0 +1,1411 @@ +# chorus-hap - Human Agent Portal Binary + +**Binary:** `chorus-hap` +**Source:** `cmd/hap/main.go` +**Status:** πŸ”Ά Beta (Terminal interface: βœ… Production, Web interface: ⏳ Stubbed) +**Purpose:** Human-friendly interface for participating in P2P agent networks + +--- + +## Overview + +`chorus-hap` (Human Agent Portal) is the executable that enables human users to participate in CHORUS P2P agent networks as first-class peers. Unlike autonomous agents, HAP provides interactive interfaces (terminal and web) for humans to collaborate with agents using the same protocols. + +### Key Features + +- βœ… **Terminal Interface**: Full-featured command-line interaction (Production) +- ⏳ **Web Interface**: Browser-based UI (Stubbed, falls back to terminal) +- βœ… **HMMM Composition**: Human-friendly reasoning message helpers +- βœ… **UCXL Browsing**: Navigate and explore context addresses +- βœ… **Decision Participation**: Vote on network decisions +- βœ… **Collaborative Sessions**: Multi-party editing coordination +- βœ… **Patch Creation**: Submit changes to codebase +- βœ… **P2P Protocols**: Same networking as autonomous agents +- βœ… **Peer Management**: View and interact with network peers + +### Human vs Agent Mode + +| Feature | chorus-agent | chorus-hap | +|---------|--------------|------------| +| **Task Execution** | Automatic | Manual approval required | +| **UI Mode** | Headless | Terminal or Web | +| **HMMM Messages** | AI-generated | Human-composed | +| **Decision Making** | Autonomous | Interactive prompts | +| **Network Role** | Autonomous peer | Human peer | +| **Use Case** | Batch processing | Oversight & collaboration | + +--- + +## Usage + +### Basic Invocation + +```bash +# With required environment variables +CHORUS_LICENSE_ID=dev-123 \ +CHORUS_AGENT_ID=human-alice \ +CHORUS_HAP_MODE=terminal \ +./chorus-hap +``` + +### Help Output + +```bash +$ ./chorus-hap --help +CHORUS-hap 0.5.0-dev + +Usage: + chorus-hap [--help] [--version] + +CHORUS Human Agent Portal - Human Interface to P2P Agent Networks + +This binary provides a human-friendly interface to participate in P2P agent +coordination networks. Humans can collaborate with autonomous agents using +the same protocols and appear as peers in the distributed network. + +Environment (common): + CHORUS_LICENSE_ID (required) + CHORUS_AGENT_ID (optional; auto-generated if empty) + CHORUS_P2P_PORT (default 9000) + CHORUS_API_PORT (default 8080) + CHORUS_HEALTH_PORT (default 8081) + CHORUS_DHT_ENABLED (default true) + CHORUS_BOOTSTRAP_PEERS (comma-separated multiaddrs) + OLLAMA_ENDPOINT (default http://localhost:11434) + +HAP-Specific Environment: + CHORUS_HAP_MODE (terminal|web, default terminal) + CHORUS_HAP_WEB_PORT (default 8082) + +Example: + CHORUS_LICENSE_ID=dev-123 \ + CHORUS_AGENT_ID=human-alice \ + CHORUS_HAP_MODE=terminal \ + CHORUS_P2P_PORT=9001 ./chorus-hap + +HAP Features: + - Human-friendly message composition + - HMMM reasoning template helpers + - UCXL context browsing + - Collaborative decision participation + - Terminal and web interface modes + - Same P2P protocols as autonomous agents +``` + +### Version Information + +```bash +$ ./chorus-hap --version +CHORUS-hap 0.5.0-dev +``` + +--- + +## Source Code Analysis + +### File: `cmd/hap/main.go` + +**Lines:** 126 +**Package:** main +**Imports:** +- `chorus/internal/hapui` - Terminal and web interface implementations +- `chorus/internal/runtime` - Shared P2P runtime infrastructure + +### main() Function Flow + +```go +func main() { + // 1. CLI Argument Handling (lines 14-58) + // - Check for --help, -h, help + // - Check for --version, -v + // - Print usage and exit early if found + + // 2. Initialize Shared Runtime (lines 60-66) + sharedRuntime, err := runtime.Initialize("hap") + if err != nil { + // Fatal error, exit 1 + } + defer sharedRuntime.Cleanup() + + // 3. Start HAP Mode (lines 68-72) + if err := startHAPMode(sharedRuntime); err != nil { + // Fatal error, exit 1 + } +} +``` + +### Execution Phases + +#### Phase 1: Early CLI Handling (lines 14-58) + +**Purpose:** Handle help/version requests without loading configuration + +**Code:** +```go +for _, a := range os.Args[1:] { + switch a { + case "--help", "-h", "help": + // Print HAP-specific help with terminal/web mode info + fmt.Printf("%s-hap %s\n\n", runtime.AppName, runtime.AppVersion) + // ... detailed usage information ... + return + case "--version", "-v": + fmt.Printf("%s-hap %s\n", runtime.AppName, runtime.AppVersion) + return + } +} +``` + +**Help Output Details (lines 17-52):** +- Describes HAP purpose and human participation model +- Lists common environment variables (same as chorus-agent) +- Lists HAP-specific variables (mode, web port) +- Shows example invocation with human-focused agent ID +- Highlights features: HMMM helpers, UCXL browsing, decision participation + +**Why Important:** Users can explore HAP capabilities without needing valid license. + +#### Phase 2: Runtime Initialization (line 61) + +**Function Call:** `runtime.Initialize("hap")` + +**What Happens:** +1. Load configuration from environment variables +2. Validate CHORUS license with KACHING server +3. Initialize AI provider (Ollama or ResetData) +4. Create P2P libp2p node (same as agent) +5. Start mDNS discovery +6. Initialize PubSub messaging +7. Setup DHT (if enabled) +8. Start election manager +9. Create task coordinator +10. Start HTTP API server +11. Start UCXI server (if enabled) +12. Initialize health checks +13. Setup SHHH sentinel + +**Returns:** `*runtime.SharedRuntime` containing all initialized components + +**Key Difference from Agent:** Mode parameter is `"hap"` instead of `"agent"`, which configures runtime for human interaction patterns. + +**See:** [internal/runtime Documentation](../internal/runtime.md) for complete initialization details + +#### Phase 3: HAP Mode Selection (lines 75-95) + +**Function:** `startHAPMode(runtime *runtime.SharedRuntime) error` + +**What Happens:** +1. Reads `CHORUS_HAP_MODE` environment variable (default: "terminal") +2. Routes to appropriate interface: + - **"terminal"**: Starts interactive command-line interface + - **"web"**: Attempts web interface (falls back to terminal if stubbed) + - **Invalid**: Returns error + +**Code:** +```go +func startHAPMode(runtime *runtime.SharedRuntime) error { + runtime.Logger.Info("πŸ‘€ Starting CHORUS Human Agent Portal (HAP)") + runtime.Logger.Info("πŸ”— Connected to P2P network as human agent") + runtime.Logger.Info("πŸ“ Ready for collaborative reasoning and decision making") + + hapMode := os.Getenv("CHORUS_HAP_MODE") + if hapMode == "" { + hapMode = "terminal" + } + + switch hapMode { + case "terminal": + return startTerminalInterface(runtime) + case "web": + return startWebInterface(runtime) + default: + return fmt.Errorf("unknown HAP mode: %s (valid: terminal, web)", hapMode) + } +} +``` + +#### Phase 4: Terminal Interface (lines 97-108) + +**Function:** `startTerminalInterface(runtime *runtime.SharedRuntime) error` + +**Implementation:** +```go +func startTerminalInterface(runtime *runtime.SharedRuntime) error { + runtime.Logger.Info("πŸ’» Starting terminal interface for human interaction") + + // Create and start the HAP terminal interface + terminal := hapui.NewTerminalInterface(runtime) + + runtime.Logger.Info("🎯 Human agent terminal interface ready") + + // Start the interactive terminal (blocks until quit) + return terminal.Start() +} +``` + +**Terminal Capabilities:** +- Interactive REPL with `hap>` prompt +- Command parsing and execution +- HMMM message composition with templates +- UCXL context browsing +- Network status monitoring +- Peer management +- Decision voting +- Collaborative session management +- Patch submission + +**See:** Terminal Interface Details section for full command reference + +#### Phase 5: Web Interface (lines 110-126) + +**Function:** `startWebInterface(runtime *runtime.SharedRuntime) error` + +**Current Status:** ⏳ **STUBBED** - Phase 3 Implementation + +**Code:** +```go +func startWebInterface(runtime *runtime.SharedRuntime) error { + runtime.Logger.Info("🌐 Starting web interface for human interaction") + + // TODO Phase 3: Implement web interface + // - HTTP server with WebSocket for real-time updates + // - Web forms for HMMM message composition + // - Context browser UI + // - Decision voting interface + + runtime.Logger.Info("⚠️ Web interface not yet implemented") + runtime.Logger.Info("πŸ”„ HAP running in stub mode - P2P connectivity established") + runtime.Logger.Info("πŸ“ Next: Implement Phase 3 web interface") + + // For now, fall back to terminal mode + return startTerminalInterface(runtime) +} +``` + +**Planned Features (Phase 3):** +- HTTP server with WebSocket for real-time updates +- Web forms for HMMM message composition +- Visual context browser UI +- Interactive decision voting interface +- Multi-user collaborative session views +- Real-time network status dashboard + +**Current Behavior:** Falls back to terminal interface with warning logs + +--- + +## Terminal Interface + +### Implementation Details + +**Source:** `internal/hapui/terminal.go` +**Lines:** ~3985 +**Status:** βœ… Production + +### Welcome Screen + +When terminal interface starts: + +``` +================================================================================ +🎭 CHORUS Human Agent Portal (HAP) - Terminal Interface +================================================================================ +Agent ID: human-alice +P2P Node: 12D3KooW...short +Connected to: 3 peers + +You are now connected to the CHORUS P2P agent network as a human participant. +You can collaborate with autonomous agents using the same protocols. +================================================================================ +``` + +### Available Commands + +#### Core Commands + +| Command | Description | Status | +|---------|-------------|--------| +| `help` | Show command reference | βœ… | +| `status` | Show network and agent status | βœ… | +| `peers` | List connected P2P peers | βœ… | +| `quit` | Exit HAP terminal | βœ… | +| `announce` | Re-announce human agent presence | βœ… | + +#### Collaborative Commands + +| Command | Description | Status | +|---------|-------------|--------| +| `hmmm` | Compose and send HMMM reasoning message | βœ… | +| `ucxl
` | Browse UCXL context address | βœ… | +| `patch` | Create and submit patches | βœ… | +| `collab` | Collaborative editing sessions | βœ… | +| `decide ` | Participate in distributed decision | βœ… | + +#### Advanced Commands + +| Command | Description | Status | +|---------|-------------|--------| +| `web` | Start web bridge for browser access | πŸ”Ά Beta | + +### Command Details + +#### `status` - Network Status + +**Usage:** `status` + +**Output:** +``` +Network Status: + Agent ID: human-alice + P2P Node ID: 12D3KooWABC... + Listening Addresses: + - /ip4/192.168.1.100/tcp/9001 + - /ip6/::1/tcp/9001 + Connected Peers: 5 + DHT Status: Active + Election: Follower (current leader: chorus-agent-3) + Active Tasks: 0/5 +``` + +**Implementation:** Queries `runtime.SharedRuntime` for live status + +#### `peers` - Peer List + +**Usage:** `peers` + +**Output:** +``` +Connected Peers (5): + 1. chorus-agent-1 (12D3KooWXYZ...) + Type: autonomous + Address: /ip4/192.168.1.101/tcp/9000 + Latency: 15ms + + 2. chorus-agent-2 (12D3KooWABC...) + Type: autonomous + Address: /ip4/192.168.1.102/tcp/9000 + Latency: 22ms + + 3. human-bob (12D3KooWDEF...) + Type: human + Address: /ip4/192.168.1.103/tcp/9001 + Latency: 8ms +``` + +**Implementation:** Enumerates `runtime.Node.ConnectedPeers()` with metadata + +#### `hmmm` - HMMM Message Composition + +**Usage:** `hmmm` + +**Interactive Flow:** + +``` +hap> hmmm + +πŸ“ Composing HMMM (Hierarchical Meta-level Model of Meaning) Message +==================================================================== + +Select HMMM message type: + 1. Observation - Share an observation about the system + 2. Question - Ask peers for input + 3. Proposal - Propose a solution or approach + 4. Objection - Raise concerns about a proposal + 5. Synthesis - Summarize discussion or findings + +Enter selection [1-5]: 3 + +Enter proposal title: Implement rate limiting for API endpoints + +Enter proposal description (end with empty line): +> We should add rate limiting to prevent abuse of public API endpoints. +> Suggested approach: token bucket algorithm with 100 req/min per IP. +> This would protect against DoS attacks while allowing normal usage. +> + +Enter affected components (comma-separated, optional): api,security + +Confidence level (0.0-1.0) [0.8]: 0.9 + +Preview: +-------- +Type: Proposal +Title: Implement rate limiting for API endpoints +Description: We should add rate limiting to prevent abuse of public API... +Components: api, security +Confidence: 0.9 +Author: human-alice + +Send this HMMM message? [y/N]: y + +βœ… HMMM message sent to network (ID: hmmm-20250930-abc123) +πŸ“‘ Published to topic: hmmm/meta-discussion/v1 +``` + +**Implementation:** +- Guided wizard for HMMM message construction +- Validation of message fields +- Publishes to `hmmm/meta-discussion/v1` PubSub topic +- Records message in local history + +#### `ucxl
` - UCXL Context Browser + +**Usage:** `ucxl ` + +**Example:** +``` +hap> ucxl ucxl://chorus/decision/20250930/task-123 + +πŸ“ UCXL Context Browser +==================================================================== +Address: ucxl://chorus/decision/20250930/task-123 + +Metadata: + Type: decision + Created: 2025-09-30 14:32:15 UTC + Creator: chorus-agent-1 + Status: completed + +Content: +{ + "task_id": "task-123", + "description": "Implement API rate limiting", + "outcome": "approved", + "votes": { + "approve": 3, + "reject": 0, + "defer": 1 + }, + "participants": [ + "chorus-agent-1", + "chorus-agent-2", + "human-alice" + ] +} + +Related Contexts: + - ucxl://chorus/task/20250930/task-123 (parent task) + - ucxl://chorus/hmmm/20250930/proposal-xyz (related discussion) + +[Press 'p' for parent, 'r' for related, 'q' to quit browser] +``` + +**Implementation:** +- Resolves UCXL address via DHT +- Parses and displays context data +- Supports navigation to related contexts +- Interactive browser for exploration + +#### `decide ` - Decision Participation + +**Usage:** `decide ` + +**Interactive Flow:** + +``` +hap> decide api-rate-limiting + +πŸ—³οΈ Decision Participation: api-rate-limiting +==================================================================== + +Decision: Should we implement rate limiting on API endpoints? +Proposed by: chorus-agent-1 (autonomous) +Status: voting +Deadline: 2025-09-30 18:00:00 UTC (2h 15m remaining) + +Current Votes: + βœ… Approve: 2 (chorus-agent-1, chorus-agent-2) + ❌ Reject: 0 + ⏸️ Defer: 1 (chorus-agent-3) + β­• Abstain: 0 + +Your vote [approve/reject/defer/abstain]: approve + +Enter reasoning (optional, end with empty line): +> Rate limiting is essential for API security. The proposed token bucket +> approach is standard and well-tested. I support implementation. +> + +Confidence level (0.0-1.0) [0.8]: 0.9 + +Confirm vote: + Decision: api-rate-limiting + Vote: approve + Reasoning: Rate limiting is essential... + Confidence: 0.9 + +Submit this vote? [y/N]: y + +βœ… Vote submitted successfully +πŸ“‘ Published to topic: chorus/decisions +πŸ“ UCXL recorded: ucxl://chorus/vote/20250930/human-alice-rate-limiting +``` + +**Implementation:** +- Fetches decision from DHT +- Displays current voting status +- Collects human vote with reasoning +- Publishes vote to network +- Records vote in UCXL + +#### `patch` - Patch Creation + +**Usage:** `patch` + +**Interactive Flow:** + +``` +hap> patch + +πŸ”§ Patch Creation & Submission +==================================================================== + +Enter patch title: Fix typo in API documentation + +Enter files affected (one per line, empty to finish): +> docs/api/endpoints.md +> + +Enter patch description: +> Fixed spelling error: "recieve" -> "receive" in authentication section +> + +Enter patch diff or file path: +> --- a/docs/api/endpoints.md +> +++ b/docs/api/endpoints.md +> @@ -15,7 +15,7 @@ +> Authentication +> -------------- +> +> -All API requests must include a valid token to recieve a response. +> +All API requests must include a valid token to receive a response. +> +> Token format: Bearer +> + +Submit patch to network? [y/N]: y + +βœ… Patch submitted (ID: patch-20250930-abc123) +πŸ“‘ Published to topic: chorus/patches +πŸ“ UCXL recorded: ucxl://chorus/patch/20250930/human-alice-typo-fix + +Awaiting peer review... +``` + +**Implementation:** +- Guided patch creation wizard +- Supports diff input or file paths +- Publishes patch to network +- Creates UCXL record for tracking +- Notifies peers for review + +#### `collab` - Collaborative Sessions + +**Usage:** `collab [start|join|status|leave]` + +**Subcommands:** + +**`collab start`** - Start new session: +``` +hap> collab start + +Starting collaborative editing session... +Session ID: collab-20250930-xyz789 +Owner: human-alice +Status: active + +Share this command with collaborators: + collab join collab-20250930-xyz789 + +Waiting for participants... +``` + +**`collab join `** - Join existing session: +``` +hap> collab join collab-20250930-xyz789 + +Joining collaborative session... +Session ID: collab-20250930-xyz789 +Owner: human-bob +Participants: human-bob, chorus-agent-5 + +βœ… Joined session successfully +Real-time editing enabled +``` + +**`collab status`** - Show session info: +``` +hap> collab status + +Active Collaborative Session: + Session ID: collab-20250930-xyz789 + Owner: human-bob + Participants: + - human-bob (owner) + - human-alice (you) + - chorus-agent-5 + Status: active + Created: 15m ago +``` + +**`collab leave`** - Exit session: +``` +hap> collab leave + +Left collaborative session collab-20250930-xyz789 +``` + +**Implementation:** +- Creates distributed editing session +- Uses PubSub for real-time updates +- Supports human and agent participants +- Tracks session state in DHT + +#### `web` - Web Bridge + +**Usage:** `web [start|stop|status]` + +**Status:** πŸ”Ά Beta + +**Subcommands:** + +**`web start`** - Start web server: +``` +hap> web start + +Starting web bridge server... +Listening on: http://localhost:8082 +WebSocket endpoint: ws://localhost:8082/ws + +Open in browser: http://localhost:8082 + +Web bridge ready for browser connections +``` + +**`web stop`** - Stop web server: +``` +hap> web stop + +Stopping web bridge server... +Web bridge stopped +``` + +**`web status`** - Show web server status: +``` +hap> web status + +Web Bridge Status: + Running: Yes + Listen Address: http://localhost:8082 + Active Connections: 2 + Uptime: 1h 23m +``` + +**Implementation:** +- Starts HTTP server with WebSocket +- Bridges terminal commands to web UI +- Allows browser-based interaction +- Maintains terminal session alongside web + +### Command Loop Structure + +**Implementation** (`internal/hapui/terminal.go` lines 131-180): + +```go +func (t *TerminalInterface) commandLoop() { + for { + fmt.Print("hap> ") + + if !t.scanner.Scan() { + break // EOF or error + } + + input := strings.TrimSpace(t.scanner.Text()) + if input == "" { + continue + } + + parts := strings.Fields(input) + command := strings.ToLower(parts[0]) + + switch command { + case "help", "h": + t.printHelp() + case "status", "s": + t.showStatus() + case "peers", "p": + t.listPeers() + case "hmmm": + t.composeHMMMMessage() + case "ucxl": + if len(parts) < 2 { + fmt.Println("Usage: ucxl
") + continue + } + t.browseUCXL(parts[1]) + case "patch": + t.createPatch() + case "collab": + t.handleCollabCommand(parts[1:]) + case "decide": + if len(parts) < 2 { + fmt.Println("Usage: decide ") + continue + } + t.participateInDecision(parts[1]) + case "web": + t.handleWebCommand(parts[1:]) + case "announce": + t.announceHumanAgent() + case "quit", "q", "exit": + t.Stop() + return + default: + fmt.Printf("Unknown command: %s (type 'help' for commands)\n", command) + } + } +} +``` + +--- + +## Configuration + +### Required Environment Variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `CHORUS_LICENSE_ID` | License key from KACHING | `dev-123` | + +### Optional Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `CHORUS_AGENT_ID` | Auto-generated | Human agent identifier (e.g., `human-alice`) | +| `CHORUS_P2P_PORT` | 9000 | libp2p listening port | +| `CHORUS_API_PORT` | 8080 | HTTP API port | +| `CHORUS_HEALTH_PORT` | 8081 | Health check port | +| `CHORUS_DHT_ENABLED` | true | Enable distributed hash table | +| `CHORUS_BOOTSTRAP_PEERS` | "" | Comma-separated multiaddrs | +| `OLLAMA_ENDPOINT` | http://localhost:11434 | Ollama API endpoint | + +### HAP-Specific Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `CHORUS_HAP_MODE` | terminal | Interface mode: `terminal` or `web` | +| `CHORUS_HAP_WEB_PORT` | 8082 | Web interface HTTP port (when mode=web) | + +### Naming Conventions + +**Human Agent IDs:** +Use human-recognizable names to distinguish from autonomous agents: + +```bash +# Good examples +CHORUS_AGENT_ID=human-alice +CHORUS_AGENT_ID=human-bob-reviewer +CHORUS_AGENT_ID=human-ops-team + +# Less clear (looks like autonomous agent) +CHORUS_AGENT_ID=agent-123 +CHORUS_AGENT_ID=chorus-worker-1 +``` + +--- + +## Runtime Behavior + +### Startup Sequence + +``` +1. Parse CLI arguments + β”œβ”€β†’ --help β†’ print help, exit 0 + β”œβ”€β†’ --version β†’ print version, exit 0 + └─→ (none) β†’ continue + +2. Initialize shared runtime + β”œβ”€β†’ Load environment configuration + β”œβ”€β†’ Validate license with KACHING + β”‚ └─→ FAIL β†’ print error, exit 1 + β”œβ”€β†’ Configure AI provider + β”œβ”€β†’ Create P2P node + β”œβ”€β†’ Start mDNS discovery + β”œβ”€β†’ Initialize PubSub + β”œβ”€β†’ Setup DHT (optional) + β”œβ”€β†’ Start election manager + β”œβ”€β†’ Create task coordinator + β”œβ”€β†’ Start HTTP API server + └─→ Initialize health checks + +3. Determine HAP mode + β”œβ”€β†’ Read CHORUS_HAP_MODE + β”œβ”€β†’ Default to "terminal" + └─→ Validate mode (terminal or web) + +4. Start appropriate interface + β”œβ”€β†’ TERMINAL: + β”‚ β”œβ”€β†’ Create TerminalInterface + β”‚ β”œβ”€β†’ Print welcome message + β”‚ β”œβ”€β†’ Show help + β”‚ β”œβ”€β†’ Announce human agent to network + β”‚ └─→ Start command loop (blocking) + β”‚ + └─→ WEB (currently stubbed): + β”œβ”€β†’ Log web mode request + β”œβ”€β†’ Log stub status warning + β”œβ”€β†’ Fall back to terminal mode + └─→ TODO: Start HTTP server with WebSocket + +5. Run until quit command or signal + β”œβ”€β†’ Terminal: wait for 'quit' command + └─→ Web: wait for SIGINT/SIGTERM + +6. Cleanup on shutdown + β”œβ”€β†’ Close active collaborative sessions + β”œβ”€β†’ Unsubscribe from PubSub topics + β”œβ”€β†’ Close P2P connections + β”œβ”€β†’ Stop HTTP servers + └─→ Exit gracefully +``` + +### Signal Handling + +| Signal | Behavior | +|--------|----------| +| SIGINT (Ctrl+C) | Graceful shutdown, close sessions | +| SIGTERM | Graceful shutdown | +| SIGHUP | Reload configuration (if applicable) | + +### Human Participation Model + +**Differences from Autonomous Agents:** + +| Aspect | chorus-agent | chorus-hap | +|--------|--------------|------------| +| **Task Assignment** | Accepts automatically | Prompts human for approval | +| **HMMM Messages** | AI-generated reasoning | Human-composed reasoning | +| **Decision Voting** | Autonomous voting | Interactive voting prompts | +| **Code Execution** | Automatic in sandbox | Manual approval required | +| **Availability** | 24/7 operation | Human working hours | +| **Network Role** | Autonomous peer | Human oversight peer | + +**Announcement to Network:** + +On startup, HAP announces human presence: + +```json +{ + "type": "peer_announcement", + "agent_id": "human-alice", + "agent_type": "human", + "capabilities": [ + "hmmm_composition", + "decision_participation", + "code_review", + "collaborative_editing" + ], + "availability": "interactive", + "timestamp": "2025-09-30T14:30:00Z" +} +``` + +Published to: `chorus/coordination/v1` topic + +**Purpose:** +- Helps autonomous agents identify human peers +- Adjusts interaction patterns (e.g., wait for human responses) +- Enables mixed human-agent collaboration + +--- + +## P2P Networking + +### Peer Discovery + +**Same as chorus-agent:** + +**mDNS (Local):** +- Discovers peers on local network +- Service name: `chorus-peer-discovery` +- Automatic peer connection + +**DHT (Global):** +- Discovers peers across networks +- Requires bootstrap peers +- Content-addressed routing + +### Topics Subscribed + +| Topic | Purpose | +|-------|---------| +| `chorus/coordination/v1` | Task coordination messages | +| `hmmm/meta-discussion/v1` | Collaborative reasoning (human participates here) | +| `chorus/election/v1` | Leader election heartbeats | +| `chorus/decisions` | Decision announcements (voting) | +| `chorus/patches` | Patch submission and review | +| `chorus/collaborative` | Collaborative editing sessions | +| `chorus/health` | Health status broadcasts | + +### Human-Specific Behaviors + +**Slower Response Times:** +HAP doesn't auto-respond to messages. Autonomous agents should: +- Wait longer for human responses +- Include timeout considerations +- Provide clear prompts/context + +**Interactive Approval:** +Tasks requiring human execution trigger interactive prompts: +``` +hap> +πŸ”” Task Assignment Request +==================================================================== +Task ID: task-123 +From: chorus-agent-2 +Description: Review security patch for API endpoints +Priority: high +Estimated Time: 30 minutes + +Accept this task? [y/N]: _ +``` + +--- + +## Health Checks + +### HTTP Endpoints + +**Liveness Probe:** +```bash +curl http://localhost:8081/healthz +# Returns: 200 OK if HAP is alive +``` + +**Readiness Probe:** +```bash +curl http://localhost:8081/ready +# Returns: 200 OK if HAP is ready (human available) +# Returns: 503 if human is away/unavailable +``` + +**Health Details:** +```bash +curl http://localhost:8081/health +# Returns JSON with: +# - P2P connectivity status +# - Interface mode (terminal/web) +# - Active sessions +# - Human availability status +``` + +### Availability Indicators + +HAP reports human availability status: + +| Status | Meaning | +|--------|---------| +| `available` | Human actively using terminal | +| `idle` | Terminal open but no recent activity | +| `away` | Extended idle period | +| `offline` | HAP closed | + +**Implementation:** +- Tracks last command timestamp +- Idle timeout: 15 minutes β†’ `idle` +- Away timeout: 1 hour β†’ `away` + +--- + +## Integration Points + +### HMMM Protocol + +**Human Participation in Meta-Discussion:** + +HAP enables humans to compose HMMM messages: + +1. **Observation**: Share insights about system behavior +2. **Question**: Ask network for input +3. **Proposal**: Suggest solutions +4. **Objection**: Raise concerns +5. **Synthesis**: Summarize findings + +**Message Format:** +```json +{ + "type": "hmmm/proposal", + "id": "hmmm-20250930-abc123", + "author": "human-alice", + "author_type": "human", + "title": "Implement rate limiting", + "description": "...", + "confidence": 0.9, + "timestamp": "2025-09-30T14:30:00Z" +} +``` + +**See:** [HMMM Protocol](../protocols/hmmm.md) + +### UCXL Context Addressing + +**Browser Support:** + +HAP provides UCXL browser for exploring contexts: +- Navigate context tree +- View related contexts +- Follow links between decisions/tasks +- Export context data + +**Example Navigation:** +``` +ucxl://chorus/task/20250930/task-123 + └─→ ucxl://chorus/decision/20250930/decision-abc + └─→ ucxl://chorus/hmmm/20250930/proposal-xyz + └─→ ucxl://chorus/patch/20250930/patch-def +``` + +**See:** [UCXL Specification](../protocols/ucxl.md) + +### Decision Voting + +**Democratic Participation:** + +Humans vote on network decisions: +- Approve/Reject/Defer/Abstain +- Provide reasoning +- Express confidence level +- Track voting history + +**Vote Weight:** +Configurable per deployment: +- Equal weight (1 human = 1 agent) +- Expert weight (humans have higher weight) +- Role-based weight + +**See:** [Decision System](../packages/decisions.md) + +--- + +## Example Deployments + +### Local Development + +```bash +#!/bin/bash +# Run HAP for local development + +export CHORUS_LICENSE_ID=dev-local-123 +export CHORUS_AGENT_ID=human-developer +export CHORUS_P2P_PORT=9001 +export CHORUS_API_PORT=8080 +export CHORUS_HEALTH_PORT=8081 +export CHORUS_HAP_MODE=terminal +export OLLAMA_ENDPOINT=http://localhost:11434 + +./chorus-hap +``` + +### Docker Container + +```dockerfile +FROM debian:bookworm-slim + +# Install runtime dependencies +RUN apt-get update && apt-get install -y \ + ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +# Copy binary +COPY chorus-hap /usr/local/bin/chorus-hap + +# Expose ports +EXPOSE 9000 8080 8081 8082 + +# HAP typically runs interactively, but can be daemonized +ENTRYPOINT ["/usr/local/bin/chorus-hap"] +``` + +```bash +docker run -it \ + --name chorus-hap-alice \ + -e CHORUS_LICENSE_ID=dev-123 \ + -e CHORUS_AGENT_ID=human-alice \ + -e CHORUS_HAP_MODE=terminal \ + -p 9001:9000 \ + -p 8080:8080 \ + -p 8081:8081 \ + chorus-hap:latest +``` + +**Note:** Interactive terminal requires `-it` flags for stdin/stdout + +### Web Mode Deployment (Future) + +```bash +# When web interface is implemented: +export CHORUS_HAP_MODE=web +export CHORUS_HAP_WEB_PORT=8082 + +./chorus-hap +# Access via http://localhost:8082 +``` + +### Multi-User Setup + +```bash +# Multiple humans can join network with different agent IDs + +# Human 1 (Alice) +CHORUS_AGENT_ID=human-alice CHORUS_P2P_PORT=9001 ./chorus-hap & + +# Human 2 (Bob) +CHORUS_AGENT_ID=human-bob CHORUS_P2P_PORT=9002 ./chorus-hap & + +# Both connect to same network via mDNS/DHT +# Both can collaborate on same decisions/tasks +``` + +--- + +## Troubleshooting + +### HAP Won't Start + +**Symptom:** HAP exits immediately with error + +**Possible Causes:** + +1. **Invalid or missing license** + ``` + ❌ Failed to initialize CHORUS HAP: license validation failed + ``` + **Fix:** Check `CHORUS_LICENSE_ID` and KACHING server connectivity + +2. **Port already in use** + ``` + ❌ Failed to initialize: bind: address already in use + ``` + **Fix:** Change `CHORUS_P2P_PORT` or kill process on port + +3. **P2P node creation failed** + ``` + ❌ Failed to create P2P node: ... + ``` + **Fix:** Check network connectivity and firewall rules + +### Terminal Not Accepting Input + +**Symptom:** `hap>` prompt doesn't respond to keyboard + +**Possible Causes:** + +1. **Running without TTY** + **Fix:** Use `-it` flags if running in Docker: `docker run -it ...` + +2. **STDIN redirected** + **Fix:** Ensure terminal has direct stdin access + +3. **Terminal encoding issues** + **Fix:** Set `LANG=en_US.UTF-8` or appropriate locale + +### No Peers Discovered + +**Symptom:** HAP shows 0 connected peers + +**Possible Causes:** + +1. **mDNS blocked by firewall** + **Fix:** Allow UDP port 5353, or use bootstrap peers + +2. **No other agents running** + **Fix:** Start at least one chorus-agent on network + +3. **Network isolation** + **Fix:** Ensure HAP can reach other peers on P2P ports + +### Web Mode Falls Back to Terminal + +**Symptom:** Web mode requested but terminal interface starts + +**Expected Behavior:** This is normal! Web interface is currently stubbed (Phase 3). + +**Log Output:** +``` +🌐 Starting web interface for human interaction +⚠️ Web interface not yet implemented +πŸ”„ HAP running in stub mode - P2P connectivity established +πŸ’» Starting terminal interface for human interaction +``` + +**Workaround:** Use terminal interface until Phase 3 implementation + +### HMMM Messages Not Received + +**Symptom:** Compose HMMM message but peers don't see it + +**Possible Causes:** + +1. **Not subscribed to topic** + **Check:** Verify PubSub subscription status + **Fix:** Restart HAP to re-subscribe + +2. **PubSub not enabled** + **Check:** `status` command shows PubSub status + **Fix:** Ensure DHT is enabled + +3. **Message formatting error** + **Check:** HAP logs for publish errors + **Fix:** Report issue with message details + +### Commands Don't Work + +**Symptom:** Type command but nothing happens + +**Possible Causes:** + +1. **Typo in command name** + **Fix:** Type `help` to see valid commands + +2. **Missing required arguments** + ``` + Usage: ucxl
+ ``` + **Fix:** Provide required arguments + +3. **Backend error (silent failure)** + **Fix:** Check HAP logs for error messages + +--- + +## Related Documentation + +- [chorus-agent](chorus-agent.md) - Autonomous agent binary +- [chorus](chorus.md) - Deprecated compatibility wrapper +- [internal/runtime](../internal/runtime.md) - Shared runtime initialization +- [internal/hapui](../internal/hapui.md) - Terminal and web interface implementations +- [HMMM Protocol](../protocols/hmmm.md) - Meta-discussion protocol +- [UCXL Specification](../protocols/ucxl.md) - Context addressing +- [Decision System](../packages/decisions.md) - Democratic voting +- [Configuration](../deployment/configuration.md) - Environment variables + +--- + +## Implementation Status + +| Feature | Status | Notes | +|---------|--------|-------| +| **Terminal Interface** | βœ… Production | Full interactive REPL | +| **Web Interface** | ⏳ Stubbed | Phase 3 implementation pending | +| **P2P Networking** | βœ… Production | Same as chorus-agent | +| **HMMM Composition** | βœ… Production | Guided wizard with templates | +| **UCXL Browser** | βœ… Production | Interactive context navigation | +| **Decision Voting** | βœ… Production | Full voting workflow | +| **Collaborative Sessions** | βœ… Production | Multi-party editing | +| **Patch Creation** | βœ… Production | Diff-based submission | +| **Web Bridge** | πŸ”Ά Beta | Experimental browser access | +| **Task Approval** | βœ… Production | Interactive task acceptance | +| **Peer Management** | βœ… Production | List and monitor peers | +| **Health Checks** | βœ… Production | Liveness & readiness | +| **Availability Tracking** | βœ… Production | Idle/away detection | +| **License Validation** | βœ… Production | KACHING integration | + +### Phase 3 Roadmap: Web Interface + +**Planned Components:** + +1. **HTTP Server with WebSocket** + - Real-time bidirectional communication + - Server-sent events for notifications + - WebSocket for collaborative sessions + +2. **Web Forms for HMMM** + - Rich text editor for message composition + - Template selection dropdown + - Preview before submission + +3. **Visual Context Browser** + - Tree view of UCXL contexts + - Graph visualization of relationships + - Search and filter capabilities + +4. **Decision Voting UI** + - Card-based decision display + - Interactive voting buttons + - Real-time vote tallies + - Voting history timeline + +5. **Dashboard** + - Network status overview + - Connected peers visualization + - Active sessions monitoring + - Personal activity feed + +6. **Multi-User Support** + - Authentication for web users + - Session management + - User presence indicators + +**Implementation Estimate:** 2-3 development sprints + +**See:** `cmd/hap/main.go` lines 114-126 for TODO details + +--- + +## Best Practices + +### Agent ID Naming + +**Recommended Format:** +```bash +# Human agents: prefix with "human-" +CHORUS_AGENT_ID=human-alice +CHORUS_AGENT_ID=human-bob-ops +CHORUS_AGENT_ID=human-security-team + +# Makes it clear in network views who is human vs autonomous +``` + +### Working Hours Configuration + +**Availability Signaling:** + +HAP automatically signals availability based on activity. For scheduled availability: + +```bash +# Set custom idle/away timeouts via configuration +# (Future enhancement) +CHORUS_HAP_IDLE_TIMEOUT=900 # 15 minutes +CHORUS_HAP_AWAY_TIMEOUT=3600 # 1 hour +``` + +### Collaborative Workflow + +**Best Practice for Human-Agent Collaboration:** + +1. **Start HAP at beginning of work session** +2. **Announce presence**: `announce` command +3. **Monitor network status**: Use `status` and `peers` regularly +4. **Participate in decisions**: Check for pending votes with `decide` +5. **Collaborate actively**: Join collaborative sessions when invited +6. **Review agent work**: Use UCXL browser to audit agent decisions +7. **Clean exit**: Use `quit` command (not Ctrl+C) for graceful shutdown + +### Security Considerations + +**Sensitive Operations:** + +Humans can override agent decisions for sensitive operations: +- Production deployments +- Database schema changes +- Security policy updates +- API versioning changes + +**Recommended Setup:** +- Use separate license for HAP (human-authorized) +- Grant HAP elevated permissions for overrides +- Require human approval for sensitive task categories + +--- + +**Last Updated:** 2025-09-30 \ No newline at end of file diff --git a/docs/comprehensive/commands/chorus.md b/docs/comprehensive/commands/chorus.md new file mode 100644 index 0000000..2c57b54 --- /dev/null +++ b/docs/comprehensive/commands/chorus.md @@ -0,0 +1,910 @@ +# chorus - Deprecated Compatibility Wrapper + +**Binary:** `chorus` +**Source:** `cmd/chorus/main.go` +**Status:** ⚠️ **DEPRECATED** (Removal planned in future version) +**Purpose:** Compatibility wrapper redirecting users to new binaries + +--- + +## Deprecation Notice + +**⚠️ THIS BINARY IS DEPRECATED AND SHOULD NOT BE USED ⚠️** + +The `chorus` binary has been **replaced** by specialized binaries: + +| Old Binary | New Binary | Purpose | +|------------|------------|---------| +| `./chorus` | `./chorus-agent` | Autonomous AI agents | +| `./chorus` | `./chorus-hap` | Human Agent Portal | + +**Migration Deadline:** This wrapper will be removed in a future version. All deployments should migrate to the new binaries immediately. + +--- + +## Overview + +The `chorus` binary is a **compatibility wrapper** that exists solely to inform users about the deprecation and guide them to the correct replacement binary. It does **not** provide any functional capabilities and will exit immediately with an error code. + +### Why Deprecated? + +**Architectural Evolution:** + +The CHORUS system evolved from a single-binary model to a multi-binary architecture to support: + +1. **Human Participation**: Enable humans to participate in agent networks as peers +2. **Separation of Concerns**: Different UIs for autonomous vs human agents +3. **Specialized Interfaces**: Terminal and web interfaces for humans +4. **Clearer Purpose**: Binary names reflect their specific roles + +**Old Architecture:** +``` +chorus (single binary) +└─→ All functionality combined +``` + +**New Architecture:** +``` +chorus-agent (autonomous operation) +β”œβ”€β†’ Headless execution +β”œβ”€β†’ Automatic task acceptance +└─→ AI-driven decision making + +chorus-hap (human interface) +β”œβ”€β†’ Terminal interface +β”œβ”€β†’ Web interface (planned) +└─→ Interactive prompts +``` + +--- + +## Usage (Deprecation Messages Only) + +### Help Output + +```bash +$ ./chorus --help +⚠️ CHORUS 0.5.0-dev - DEPRECATED BINARY + +This binary has been replaced by specialized binaries: + +πŸ€– chorus-agent - Autonomous AI agent for task coordination +πŸ‘€ chorus-hap - Human Agent Portal for human participation + +Migration Guide: + OLD: ./chorus + NEW: ./chorus-agent (for autonomous agents) + ./chorus-hap (for human agents) + +Why this change? + - Enables human participation in agent networks + - Better separation of concerns + - Specialized interfaces for different use cases + - Shared P2P infrastructure with different UIs + +For help with the new binaries: + ./chorus-agent --help + ./chorus-hap --help +``` + +### Version Output + +```bash +$ ./chorus --version +CHORUS 0.5.0-dev (DEPRECATED) +``` + +### Direct Execution (Error) + +```bash +$ ./chorus +⚠️ DEPRECATION WARNING: The 'chorus' binary is deprecated! + +This binary has been replaced with specialized binaries: + πŸ€– chorus-agent - For autonomous AI agents + πŸ‘€ chorus-hap - For human agent participation + +Please use one of the new binaries instead: + ./chorus-agent --help + ./chorus-hap --help + +This wrapper will be removed in a future version. + +# Exit code: 1 +``` + +**Important:** The binary exits with code **1** to prevent accidental use in scripts or deployments. + +--- + +## Source Code Analysis + +### File: `cmd/chorus/main.go` + +**Lines:** 63 +**Package:** main +**Imports:** +- `chorus/internal/runtime` - Only for version constants + +**Purpose:** Print deprecation messages and exit + +### Complete Source Breakdown + +#### Lines 1-9: Package Declaration and Imports + +```go +package main + +import ( + "fmt" + "os" + + "chorus/internal/runtime" +) +``` + +**Note:** Minimal imports since binary only prints messages. + +#### Lines 10-12: Deprecation Comment + +```go +// DEPRECATED: This binary is deprecated in favor of chorus-agent and chorus-hap +// This compatibility wrapper redirects users to the appropriate new binary +``` + +**Documentation:** Clear deprecation notice in code comments. + +#### Lines 13-29: main() Function + +```go +func main() { + // Early CLI handling: print help/version/deprecation notice + for _, a := range os.Args[1:] { + switch a { + case "--help", "-h", "help": + printDeprecationHelp() + return + case "--version", "-v": + fmt.Printf("%s %s (DEPRECATED)\n", runtime.AppName, runtime.AppVersion) + return + } + } + + // Print deprecation warning for direct execution + printDeprecationWarning() + os.Exit(1) +} +``` + +**Flow:** + +1. **CLI Argument Parsing** (lines 15-24): + - Check for `--help`, `-h`, `help`: Print help and exit 0 + - Check for `--version`, `-v`: Print version with deprecation tag and exit 0 + - No arguments or unknown arguments: Continue to deprecation warning + +2. **Deprecation Warning** (lines 26-28): + - Print warning message to stderr + - Exit with code 1 (error) + +**Exit Codes:** + +| Scenario | Exit Code | Purpose | +|----------|-----------|---------| +| `--help` | 0 | Normal help display | +| `--version` | 0 | Normal version display | +| Direct execution | 1 | Prevent accidental use | +| Unknown arguments | 1 | Force user to read deprecation message | + +#### Lines 31-52: printDeprecationHelp() + +```go +func printDeprecationHelp() { + fmt.Printf("⚠️ %s %s - DEPRECATED BINARY\n\n", runtime.AppName, runtime.AppVersion) + fmt.Println("This binary has been replaced by specialized binaries:") + fmt.Println() + fmt.Println("πŸ€– chorus-agent - Autonomous AI agent for task coordination") + fmt.Println("πŸ‘€ chorus-hap - Human Agent Portal for human participation") + fmt.Println() + fmt.Println("Migration Guide:") + fmt.Println(" OLD: ./chorus") + fmt.Println(" NEW: ./chorus-agent (for autonomous agents)") + fmt.Println(" ./chorus-hap (for human agents)") + fmt.Println() + fmt.Println("Why this change?") + fmt.Println(" - Enables human participation in agent networks") + fmt.Println(" - Better separation of concerns") + fmt.Println(" - Specialized interfaces for different use cases") + fmt.Println(" - Shared P2P infrastructure with different UIs") + fmt.Println() + fmt.Println("For help with the new binaries:") + fmt.Println(" ./chorus-agent --help") + fmt.Println(" ./chorus-hap --help") +} +``` + +**Content Breakdown:** + +| Section | Lines | Purpose | +|---------|-------|---------| +| Header | 32-33 | Show deprecation status with warning emoji | +| Replacement Info | 34-36 | List new binaries and their purposes | +| Migration Guide | 37-41 | Show old vs new commands | +| Rationale | 42-46 | Explain why change was made | +| Next Steps | 47-51 | Direct users to help for new binaries | + +**Design:** User-friendly guidance with: +- Clear visual indicators (emojis) +- Side-by-side comparison (OLD/NEW) +- Contextual explanations (Why?) +- Actionable next steps (--help commands) + +#### Lines 54-63: printDeprecationWarning() + +```go +func printDeprecationWarning() { + fmt.Fprintf(os.Stderr, "⚠️ DEPRECATION WARNING: The 'chorus' binary is deprecated!\n\n") + fmt.Fprintf(os.Stderr, "This binary has been replaced with specialized binaries:\n") + fmt.Fprintf(os.Stderr, " πŸ€– chorus-agent - For autonomous AI agents\n") + fmt.Fprintf(os.Stderr, " πŸ‘€ chorus-hap - For human agent participation\n\n") + fmt.Fprintf(os.Stderr, "Please use one of the new binaries instead:\n") + fmt.Fprintf(os.Stderr, " ./chorus-agent --help\n") + fmt.Fprintf(os.Stderr, " ./chorus-hap --help\n\n") + fmt.Fprintf(os.Stderr, "This wrapper will be removed in a future version.\n") +} +``` + +**Key Differences from Help:** + +| Aspect | printDeprecationHelp() | printDeprecationWarning() | +|--------|------------------------|---------------------------| +| **Output Stream** | stdout | **stderr** | +| **Verbosity** | Detailed explanation | Brief warning | +| **Tone** | Educational | Urgent | +| **Exit Code** | 0 | **1** | +| **Context** | User requested help | Accidental execution | + +**Why stderr?** + +- Ensures warning appears in logs +- Distinguishes error from normal output +- Prevents piping warning into scripts +- Signals abnormal execution + +**Why brief?** + +- User likely expected normal execution +- Quick redirection to correct binary +- Reduces noise in automated systems +- Clear that this is an error condition + +--- + +## Migration Guide + +### For Deployment Scripts + +**Old Script:** +```bash +#!/bin/bash +# DEPRECATED - DO NOT USE + +export CHORUS_LICENSE_ID=prod-123 +export CHORUS_AGENT_ID=chorus-worker-1 + +# This will fail with exit code 1 +./chorus +``` + +**New Script (Autonomous Agent):** +```bash +#!/bin/bash +# Updated for chorus-agent + +export CHORUS_LICENSE_ID=prod-123 +export CHORUS_AGENT_ID=chorus-worker-1 +export CHORUS_P2P_PORT=9000 + +# Use new agent binary +./chorus-agent +``` + +**New Script (Human Agent):** +```bash +#!/bin/bash +# Updated for chorus-hap + +export CHORUS_LICENSE_ID=prod-123 +export CHORUS_AGENT_ID=human-alice +export CHORUS_HAP_MODE=terminal + +# Use new HAP binary +./chorus-hap +``` + +### For Docker Deployments + +**Old Dockerfile:** +```dockerfile +FROM debian:bookworm-slim +COPY chorus /usr/local/bin/chorus +ENTRYPOINT ["/usr/local/bin/chorus"] # DEPRECATED +``` + +**New Dockerfile (Agent):** +```dockerfile +FROM debian:bookworm-slim +RUN apt-get update && apt-get install -y ca-certificates docker.io +COPY chorus-agent /usr/local/bin/chorus-agent +ENTRYPOINT ["/usr/local/bin/chorus-agent"] +``` + +**New Dockerfile (HAP):** +```dockerfile +FROM debian:bookworm-slim +RUN apt-get update && apt-get install -y ca-certificates +COPY chorus-hap /usr/local/bin/chorus-hap +ENTRYPOINT ["/usr/local/bin/chorus-hap"] +``` + +### For Docker Compose + +**Old docker-compose.yml:** +```yaml +services: + chorus: # DEPRECATED + image: chorus:latest + command: /chorus # Will fail +``` + +**New docker-compose.yml (Agent):** +```yaml +services: + chorus-agent: + image: chorus-agent:latest + command: /usr/local/bin/chorus-agent + environment: + - CHORUS_LICENSE_ID=${CHORUS_LICENSE_ID} +``` + +**New docker-compose.yml (HAP):** +```yaml +services: + chorus-hap: + image: chorus-hap:latest + command: /usr/local/bin/chorus-hap + stdin_open: true # Required for terminal interface + tty: true + environment: + - CHORUS_LICENSE_ID=${CHORUS_LICENSE_ID} + - CHORUS_HAP_MODE=terminal +``` + +### For Systemd Services + +**Old Service File:** `/etc/systemd/system/chorus.service` +```ini +[Unit] +Description=CHORUS Agent (DEPRECATED) + +[Service] +ExecStart=/usr/local/bin/chorus # Will fail +Restart=always + +[Install] +WantedBy=multi-user.target +``` + +**New Service File:** `/etc/systemd/system/chorus-agent.service` +```ini +[Unit] +Description=CHORUS Autonomous Agent +After=network.target docker.service + +[Service] +Type=simple +User=chorus +EnvironmentFile=/etc/chorus/agent.env +ExecStart=/usr/local/bin/chorus-agent +Restart=on-failure +RestartSec=10s + +[Install] +WantedBy=multi-user.target +``` + +**Migration Steps:** +```bash +# Stop old service +sudo systemctl stop chorus +sudo systemctl disable chorus + +# Install new service +sudo cp chorus-agent.service /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable chorus-agent +sudo systemctl start chorus-agent +``` + +### For CI/CD Pipelines + +**Old Pipeline (GitLab CI):** +```yaml +build: + script: + - go build -o chorus ./cmd/chorus # DEPRECATED + - ./chorus --version +``` + +**New Pipeline:** +```yaml +build: + script: + - make build-agent # Builds chorus-agent + - make build-hap # Builds chorus-hap + - ./build/chorus-agent --version + - ./build/chorus-hap --version +``` + +### For Kubernetes Deployments + +**Old Deployment:** +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: chorus # DEPRECATED +spec: + template: + spec: + containers: + - name: chorus + image: chorus:latest + command: ["/chorus"] # Will fail +``` + +**New Deployment:** +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: chorus-agent +spec: + template: + spec: + containers: + - name: chorus-agent + image: chorus-agent:latest + command: ["/usr/local/bin/chorus-agent"] + env: + - name: CHORUS_LICENSE_ID + valueFrom: + secretKeyRef: + name: chorus-secrets + key: license-id +``` + +--- + +## Build Process + +### Current Makefile Targets + +The CHORUS Makefile provides migration-friendly targets: + +```makefile +# Build all binaries +make all +β”œβ”€β†’ make build-agent # Builds chorus-agent (recommended) +β”œβ”€β†’ make build-hap # Builds chorus-hap (recommended) +└─→ make build-compat # Builds chorus (deprecated wrapper) +``` + +### Building Individual Binaries + +**Autonomous Agent:** +```bash +make build-agent +# Output: build/chorus-agent +``` + +**Human Agent Portal:** +```bash +make build-hap +# Output: build/chorus-hap +``` + +**Deprecated Wrapper:** +```bash +make build-compat +# Output: build/chorus (for compatibility only) +``` + +### Why Keep the Deprecated Binary? + +**Reasons to Build chorus:** + +1. **Gradual Migration**: Allows staged rollout of new binaries +2. **Error Detection**: Catches deployments still using old binary +3. **User Guidance**: Provides migration instructions at runtime +4. **CI/CD Compatibility**: Prevents hard breaks in existing pipelines + +**Planned Removal:** + +The `chorus` binary and `make build-compat` target will be removed in: +- **Version:** 1.0.0 +- **Timeline:** After all known deployments migrate +- **Warning Period:** At least 3 minor versions (e.g., 0.5 β†’ 0.6 β†’ 0.7 β†’ 1.0) + +--- + +## Troubleshooting + +### Script Fails with "DEPRECATION WARNING" + +**Symptom:** +```bash +$ ./deploy.sh +⚠️ DEPRECATION WARNING: The 'chorus' binary is deprecated! +... +# Script exits with error +``` + +**Cause:** Script uses old `./chorus` binary + +**Fix:** +```bash +# Update script to use chorus-agent +sed -i 's|./chorus|./chorus-agent|g' deploy.sh + +# Or update to chorus-hap for human agents +sed -i 's|./chorus|./chorus-hap|g' deploy.sh +``` + +### Docker Container Exits Immediately + +**Symptom:** +```bash +$ docker run chorus:latest +⚠️ DEPRECATION WARNING: The 'chorus' binary is deprecated! +# Container exits with code 1 +``` + +**Cause:** Container uses deprecated binary + +**Fix:** Rebuild image with correct binary: +```dockerfile +# Old +COPY chorus /usr/local/bin/chorus + +# New +COPY chorus-agent /usr/local/bin/chorus-agent +ENTRYPOINT ["/usr/local/bin/chorus-agent"] +``` + +### Systemd Service Fails to Start + +**Symptom:** +```bash +$ sudo systemctl status chorus +● chorus.service - CHORUS Agent + Active: failed (Result: exit-code) + Main PID: 12345 (code=exited, status=1/FAILURE) +``` + +**Cause:** Service configured to run deprecated binary + +**Fix:** Create new service file: +```bash +# Disable old service +sudo systemctl stop chorus +sudo systemctl disable chorus + +# Create new service +sudo cp chorus-agent.service /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable chorus-agent +sudo systemctl start chorus-agent +``` + +### CI Build Succeeds but Tests Fail + +**Symptom:** +```bash +$ ./chorus --version +CHORUS 0.5.0-dev (DEPRECATED) +# Tests that try to run ./chorus fail +``` + +**Cause:** Tests invoke deprecated binary + +**Fix:** Update test commands: +```bash +# Old test +./chorus --help + +# New test +./chorus-agent --help +``` + +### Can't Find Replacement Binary + +**Symptom:** +```bash +$ ./chorus-agent +bash: ./chorus-agent: No such file or directory +``` + +**Cause:** New binaries not built or installed + +**Fix:** +```bash +# Build new binaries +make build-agent +make build-hap + +# Binaries created in build/ directory +ls -la build/chorus-* + +# Install to system +sudo cp build/chorus-agent /usr/local/bin/ +sudo cp build/chorus-hap /usr/local/bin/ +``` + +--- + +## Migration Checklist + +### Pre-Migration Assessment + +- [ ] **Inventory Deployments**: List all places `chorus` binary is used + - Production servers + - Docker images + - Kubernetes deployments + - CI/CD pipelines + - Developer machines + - Documentation + +- [ ] **Identify Binary Types**: Determine which replacement is needed + - Autonomous operation β†’ `chorus-agent` + - Human interaction β†’ `chorus-hap` + - Mixed use β†’ Both binaries needed + +- [ ] **Review Configuration**: Check environment variables + - `CHORUS_AGENT_ID` naming conventions + - HAP-specific variables (`CHORUS_HAP_MODE`) + - Port assignments (avoid conflicts) + +### Migration Execution + +- [ ] **Build New Binaries** + ```bash + make build-agent + make build-hap + ``` + +- [ ] **Update Docker Images** + - Modify Dockerfile to use new binaries + - Rebuild and tag images + - Push to registry + +- [ ] **Update Deployment Configs** + - docker-compose.yml + - kubernetes manifests + - systemd service files + - deployment scripts + +- [ ] **Test in Staging** + - Deploy new binaries to staging environment + - Verify P2P connectivity + - Test agent/HAP functionality + - Validate health checks + +- [ ] **Update CI/CD Pipelines** + - Build configurations + - Test scripts + - Deployment scripts + - Rollback procedures + +- [ ] **Deploy to Production** + - Rolling deployment (one node at a time) + - Monitor logs for deprecation warnings + - Verify peer discovery still works + - Check metrics and health endpoints + +- [ ] **Update Documentation** + - README files + - Deployment guides + - Runbooks + - Architecture diagrams + +### Post-Migration Verification + +- [ ] **Verify No Deprecation Warnings** + ```bash + # Check logs for deprecation messages + journalctl -u chorus-agent | grep DEPRECATION + # Should return no results + ``` + +- [ ] **Confirm Binary Versions** + ```bash + ./chorus-agent --version + ./chorus-hap --version + # Should show correct version without (DEPRECATED) + ``` + +- [ ] **Test Functionality** + - [ ] P2P peer discovery works + - [ ] Tasks execute successfully (agents) + - [ ] Terminal interface works (HAP) + - [ ] Health checks pass + - [ ] Metrics collected + +- [ ] **Remove Old Binary** + ```bash + # After confirming everything works + rm /usr/local/bin/chorus + ``` + +- [ ] **Clean Up Old Configs** + - Remove old systemd service files + - Delete old Docker images + - Archive old deployment scripts + +--- + +## Comparison with New Binaries + +### Feature Comparison + +| Feature | chorus (deprecated) | chorus-agent | chorus-hap | +|---------|---------------------|--------------|------------| +| **Functional** | ❌ No | βœ… Yes | βœ… Yes | +| **P2P Networking** | ❌ N/A | βœ… Yes | βœ… Yes | +| **Task Execution** | ❌ N/A | βœ… Automatic | βœ… Interactive | +| **UI Mode** | ❌ N/A | Headless | Terminal/Web | +| **Purpose** | Deprecation notice | Autonomous agent | Human interface | +| **Exit Code** | 1 (error) | 0 (normal) | 0 (normal) | +| **Runtime** | Immediate exit | Long-running | Long-running | + +### Size Comparison + +| Binary | Size | Notes | +|--------|------|-------| +| `chorus` | ~2 MB | Minimal (messages only) | +| `chorus-agent` | ~25 MB | Full functionality + dependencies | +| `chorus-hap` | ~28 MB | Full functionality + UI components | + +**Why is chorus smaller?** +- No P2P libraries linked +- No task execution engine +- No AI provider integrations +- Only runtime constants imported + +### Command Comparison + +**chorus (deprecated):** +```bash +./chorus --help # Prints deprecation help +./chorus --version # Prints version with (DEPRECATED) +./chorus # Prints warning, exits 1 +``` + +**chorus-agent:** +```bash +./chorus-agent --help # Prints agent help +./chorus-agent --version # Prints version +./chorus-agent # Runs autonomous agent +``` + +**chorus-hap:** +```bash +./chorus-hap --help # Prints HAP help +./chorus-hap --version # Prints version +./chorus-hap # Runs human interface +``` + +--- + +## Related Documentation + +- [chorus-agent](chorus-agent.md) - Autonomous agent binary (REPLACEMENT) +- [chorus-hap](chorus-hap.md) - Human Agent Portal binary (REPLACEMENT) +- [internal/runtime](../internal/runtime.md) - Shared runtime initialization +- [Migration Guide](../deployment/migration-v0.5.md) - Detailed migration instructions +- [Deployment](../deployment/docker.md) - Docker deployment guide + +--- + +## Implementation Status + +| Feature | Status | Notes | +|---------|--------|-------| +| Deprecation Messages | βœ… Implemented | Help and warning outputs | +| Exit Code 1 | βœ… Implemented | Prevents accidental use | +| Version Tagging | βœ… Implemented | Shows (DEPRECATED) | +| Guidance to New Binaries | βœ… Implemented | Clear migration instructions | +| **Removal Planned** | ⏳ Scheduled | Version 1.0.0 | + +### Removal Timeline + +| Version | Action | Date | +|---------|--------|------| +| 0.5.0 | Deprecated, wrapper implemented | 2025-09-30 | +| 0.6.0 | Warning messages in logs | TBD | +| 0.7.0 | Final warning before removal | TBD | +| 1.0.0 | **Binary removed entirely** | TBD | + +**Recommendation:** Migrate immediately. Do not wait for removal. + +--- + +## FAQ + +### Q: Can I still use `./chorus`? + +**A:** Technically you can build it, but it does nothing except print deprecation warnings and exit with error code 1. You should migrate to `chorus-agent` or `chorus-hap` immediately. + +### Q: Will `chorus` ever be restored? + +**A:** No. The architecture has permanently moved to specialized binaries. The `chorus` wrapper exists only to guide users to the correct replacement. + +### Q: What if I need both agent and HAP functionality? + +**A:** Run both binaries separately: +```bash +# Terminal 1: Run autonomous agent +./chorus-agent & + +# Terminal 2: Run human interface +./chorus-hap +``` + +Both can join the same P2P network and collaborate. + +### Q: How do I test if my deployment uses the deprecated binary? + +**A:** Check for deprecation warnings in logs: +```bash +# Grep for deprecation messages +journalctl -u chorus | grep "DEPRECATION WARNING" +docker logs 2>&1 | grep "DEPRECATION WARNING" + +# If found, migration is needed +``` + +### Q: Is there a compatibility mode? + +**A:** No. The `chorus` binary is intentionally non-functional to force migration. There is no compatibility mode. + +### Q: What about shell scripts that call `./chorus`? + +**A:** Update them to call `./chorus-agent` or `./chorus-hap`. Use `sed` for bulk updates: +```bash +# Update all scripts in directory +find . -type f -name "*.sh" -exec sed -i 's|./chorus[^-]|./chorus-agent|g' {} + +``` + +### Q: Will old Docker images still work? + +**A:** No. Docker images built with the `chorus` binary will fail at runtime with deprecation warnings. Rebuild images with new binaries. + +### Q: Can I delay migration? + +**A:** You can delay, but the wrapper will be removed in version 1.0.0. Migrate now to avoid emergency updates later. + +### Q: Where can I get help with migration? + +**A:** See: +- [Migration Guide](../deployment/migration-v0.5.md) - Detailed migration steps +- [chorus-agent Documentation](chorus-agent.md) - Agent replacement details +- [chorus-hap Documentation](chorus-hap.md) - HAP replacement details + +--- + +**Last Updated:** 2025-09-30 + +**Deprecation Status:** Active deprecation since version 0.5.0 + +**Removal Target:** Version 1.0.0 \ No newline at end of file