Created complete documentation infrastructure with master index and detailed command-line tool documentation. Documentation Structure: - docs/comprehensive/README.md - Master index with navigation - docs/comprehensive/architecture/README.md - System architecture overview - docs/comprehensive/commands/chorus-agent.md - Autonomous agent binary (✅ Production) - docs/comprehensive/commands/chorus-hap.md - Human Agent Portal (🔶 Beta) - docs/comprehensive/commands/chorus.md - Deprecated wrapper (⚠️ Deprecated) Coverage Statistics: - 3 command binaries fully documented (3,056 lines, ~14,500 words) - Complete source code analysis with line numbers - Configuration reference for all environment variables - Runtime behavior and execution flows - P2P networking details - Health checks and monitoring - Example deployments (local, Docker, Swarm) - Troubleshooting guides - Cross-references between docs Key Features Documented: - Container-first architecture - P2P mesh networking - Democratic leader election - Docker sandbox execution - HMMM collaborative reasoning - UCXL decision publishing - DHT encrypted storage - Multi-layer security - Human-agent collaboration Implementation Status Tracking: - ✅ Production features marked - 🔶 Beta features identified - ⏳ Stubbed components noted - ⚠️ Deprecated code flagged Next Phase: Package documentation (30+ packages in pkg/) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
19 KiB
chorus-agent - Autonomous Agent Binary
Binary: chorus-agent
Source: cmd/agent/main.go
Status: ✅ Production
Purpose: Autonomous AI agent for P2P task coordination
Overview
chorus-agent is the primary executable for running autonomous AI agents in the CHORUS system. Agents participate in peer-to-peer networks, execute tasks in isolated Docker sandboxes, collaborate with other agents via HMMM protocol, and maintain distributed state through DHT storage.
Key Features
- ✅ Autonomous Operation: Executes tasks without human intervention
- ✅ P2P Networking: Participates in distributed mesh network
- ✅ Docker Sandboxing: Isolated code execution environments
- ✅ HMMM Reasoning: Collaborative meta-discussion protocol
- ✅ DHT Storage: Encrypted distributed data storage
- ✅ UCXL Publishing: Immutable decision recording
- ✅ Democratic Elections: Participates in leader election
- ✅ Health Monitoring: Self-reporting health status
Usage
Basic Invocation
# With required environment variables
CHORUS_LICENSE_ID=dev-123 \
CHORUS_AGENT_ID=chorus-agent-1 \
./chorus-agent
Help Output
$ ./chorus-agent --help
CHORUS-agent 0.5.0-dev (build: abc123, 2025-09-30)
Usage:
chorus-agent [--help] [--version]
CHORUS Autonomous Agent - P2P Task Coordination
This binary runs autonomous AI agents that participate in P2P task coordination,
collaborative reasoning via HMMM, and distributed decision making.
Environment (common):
CHORUS_LICENSE_ID (required)
CHORUS_AGENT_ID (optional; auto-generated if empty)
CHORUS_P2P_PORT (default 9000)
CHORUS_API_PORT (default 8080)
CHORUS_HEALTH_PORT (default 8081)
CHORUS_DHT_ENABLED (default true)
CHORUS_BOOTSTRAP_PEERS (comma-separated multiaddrs)
OLLAMA_ENDPOINT (default http://localhost:11434)
Example:
CHORUS_LICENSE_ID=dev-123 \
CHORUS_AGENT_ID=chorus-agent-1 \
CHORUS_P2P_PORT=9000 CHORUS_API_PORT=8080 ./chorus-agent
Agent Features:
- Autonomous task execution
- P2P mesh networking
- HMMM collaborative reasoning
- DHT encrypted storage
- UCXL context addressing
- Democratic leader election
- Health monitoring
Version Information
$ ./chorus-agent --version
CHORUS-agent 0.5.0-dev (build: abc123, 2025-09-30)
Source Code Analysis
File: cmd/agent/main.go
Lines: 79 Package: main Imports:
chorus/internal/runtime- Shared P2P runtime infrastructure
Build-Time Variables
// Lines 11-16
var (
version = "0.5.0-dev"
commitHash = "unknown"
buildDate = "unknown"
)
Set via ldflags:
go build -ldflags "-X main.version=1.0.0 -X main.commitHash=$(git rev-parse --short HEAD) -X main.buildDate=$(date -u +%Y-%m-%d)"
main() Function Flow
func main() {
// 1. CLI Argument Handling (lines 19-59)
// - Check for --help, -h, help
// - Check for --version, -v
// - Print usage and exit early if found
// 2. Set Build Information (lines 61-64)
runtime.AppVersion = version
runtime.AppCommitHash = commitHash
runtime.AppBuildDate = buildDate
// 3. Initialize Shared Runtime (lines 66-72)
sharedRuntime, err := runtime.Initialize("agent")
if err != nil {
// Fatal error, exit 1
}
defer sharedRuntime.Cleanup()
// 4. Start Agent Mode (lines 74-78)
if err := sharedRuntime.StartAgentMode(); err != nil {
// Fatal error, exit 1
}
}
Execution Phases
Phase 1: Early CLI Handling (lines 19-59)
Purpose: Handle help/version requests without loading configuration
Code:
for _, a := range os.Args[1:] {
switch a {
case "--help", "-h", "help":
// Print detailed help message
fmt.Printf("%s-agent %s (build: %s, %s)\n\n", runtime.AppName, version, commitHash, buildDate)
// ... usage information ...
return
case "--version", "-v":
fmt.Printf("%s-agent %s (build: %s, %s)\n", runtime.AppName, version, commitHash, buildDate)
return
}
}
Why Important: Allows users to get help without needing valid license or configuration.
Phase 2: Runtime Initialization (line 67)
Function Call: runtime.Initialize("agent")
What Happens:
- Load configuration from environment variables
- Validate CHORUS license with KACHING server
- Initialize AI provider (Ollama or ResetData)
- Create P2P libp2p node
- Start mDNS discovery
- Initialize PubSub messaging
- Setup DHT (if enabled)
- Start election manager
- Create task coordinator
- Start HTTP API server
- Start UCXI server (if enabled)
- Initialize health checks
- Setup SHHH sentinel (secrets detection)
- Configure metrics collection
Returns: *runtime.SharedRuntime containing all initialized components
See: internal/runtime Documentation for complete initialization details
Phase 3: Agent Mode Activation (line 75)
Function Call: sharedRuntime.StartAgentMode()
What Happens:
- Agent registers itself as available for tasks
- Begins listening for task assignments via PubSub
- Starts autonomous task execution loops
- Enables automatic decision making
- Activates HMMM meta-discussion participation
- Begins heartbeat broadcasting for election
Implementation: See internal/runtime/agent_support.go
Behavior Differences from HAP:
- Agent: Automatically accepts and executes tasks
- HAP: Prompts human for task approval
Configuration
Required Environment Variables
| Variable | Description | Example |
|---|---|---|
CHORUS_LICENSE_ID |
License key from KACHING | dev-123 |
Optional Environment Variables
| Variable | Default | Description |
|---|---|---|
CHORUS_AGENT_ID |
Auto-generated | Unique agent identifier |
CHORUS_P2P_PORT |
9000 | libp2p listening port |
CHORUS_API_PORT |
8080 | HTTP API port |
CHORUS_HEALTH_PORT |
8081 | Health check port |
CHORUS_DHT_ENABLED |
true | Enable distributed hash table |
CHORUS_BOOTSTRAP_PEERS |
"" | Comma-separated multiaddrs |
OLLAMA_ENDPOINT |
http://localhost:11434 | Ollama API endpoint |
Role-Based Configuration
| Variable | Default | Description |
|---|---|---|
CHORUS_AGENT_ROLE |
"" | Agent role (admin, developer, reviewer) |
CHORUS_AGENT_EXPERTISE |
"" | Comma-separated expertise areas |
CHORUS_AGENT_REPORTS_TO |
"" | Supervisor agent ID |
CHORUS_AGENT_SPECIALIZATION |
"general" | Task specialization |
CHORUS_AGENT_MAX_TASKS |
3 | Max concurrent tasks |
AI Provider Configuration
Ollama (Default)
export CHORUS_AI_PROVIDER=ollama
export OLLAMA_ENDPOINT=http://192.168.1.72:11434
ResetData
export CHORUS_AI_PROVIDER=resetdata
export RESETDATA_API_KEY=your-api-key-here
export RESETDATA_BASE_URL=https://api.resetdata.ai
export RESETDATA_MODEL=claude-3-5-sonnet-20250930
Assignment Loading
Agents can load dynamic configuration from WHOOSH:
export ASSIGN_URL=https://whoosh.example.com/api/assignments/agent-123.json
When configured, agents:
- Fetch assignment JSON on startup
- Merge with environment config
- Listen for SIGHUP to reload
- Update configuration without restart
See: Configuration Management for assignment schema
Runtime Behavior
Startup Sequence
1. Parse CLI arguments
├─→ --help → print help, exit 0
├─→ --version → print version, exit 0
└─→ (none) → continue
2. Set build information in runtime package
3. Initialize shared runtime
├─→ Load environment configuration
├─→ Validate license with KACHING
│ └─→ FAIL → print error, exit 1
├─→ Configure AI provider
├─→ Create P2P node
├─→ Start mDNS discovery
├─→ Initialize PubSub
├─→ Setup DHT (optional)
├─→ Start election manager
├─→ Create task coordinator
├─→ Start HTTP API server
└─→ Initialize health checks
4. Start agent mode
├─→ Register as available agent
├─→ Join task coordination topics
├─→ Begin heartbeat broadcasting
├─→ Enable autonomous task execution
└─→ Activate HMMM participation
5. Run until signal (SIGINT, SIGTERM)
6. Cleanup on shutdown
├─→ Stop accepting new tasks
├─→ Complete in-flight tasks
├─→ Close P2P connections
├─→ Flush DHT cache
├─→ Stop HTTP servers
└─→ Exit gracefully
Signal Handling
| Signal | Behavior |
|---|---|
| SIGINT | Graceful shutdown (complete current tasks) |
| SIGTERM | Graceful shutdown (complete current tasks) |
| SIGHUP | Reload configuration from ASSIGN_URL |
Task Execution Loop
Once in agent mode:
Loop Forever:
│
├─→ Listen for tasks on PubSub topic "chorus/tasks"
│
├─→ Task received:
│ ├─→ Check agent availability (< max tasks)
│ ├─→ Check task matches specialization
│ └─→ Accept or decline
│
├─→ Task accepted:
│ ├─→ Increment active task count
│ ├─→ Log task start to Hypercore
│ ├─→ Invoke execution engine
│ │ ├─→ Select Docker image based on language
│ │ ├─→ Create sandbox container
│ │ ├─→ Execute commands via Docker Exec API
│ │ ├─→ Stream output
│ │ ├─→ Monitor resource usage
│ │ └─→ Capture results
│ ├─→ Generate task summary
│ ├─→ Create UCXL decision record
│ ├─→ Publish decision to DHT
│ ├─→ Broadcast completion on PubSub
│ ├─→ Decrement active task count
│ └─→ Log task completion to Hypercore
│
└─→ Continue listening
See: Task Execution Engine for execution details
P2P Networking
Peer Discovery
mDNS (Local):
- Discovers peers on local network
- Service name:
chorus-peer-discovery - No configuration required
- Automatic peer connection
DHT (Global):
- Discovers peers across networks
- Requires bootstrap peers
- Content-addressed routing
- Kademlia-based DHT
Bootstrap Peers:
export CHORUS_BOOTSTRAP_PEERS="/ip4/192.168.1.100/tcp/9000/p2p/12D3KooWABC...,/ip4/192.168.1.101/tcp/9000/p2p/12D3KooWXYZ..."
Topics Subscribed
| Topic | Purpose |
|---|---|
chorus/coordination/v1 |
Task coordination messages |
hmmm/meta-discussion/v1 |
Collaborative reasoning |
chorus/election/v1 |
Leader election heartbeats |
chorus/decisions |
Decision announcements |
chorus/health |
Health status broadcasts |
Role-Based Topics (Optional)
If CHORUS_AGENT_ROLE is set, agent also joins:
| Topic | Purpose |
|---|---|
chorus/role/{role} |
Role-specific coordination |
chorus/expertise/{expertise} |
Expertise-based routing |
chorus/reports/{supervisor} |
Reporting hierarchy |
Health Checks
HTTP Endpoints
Liveness Probe:
curl http://localhost:8081/healthz
# Returns: 200 OK if agent is alive
Readiness Probe:
curl http://localhost:8081/ready
# Returns: 200 OK if agent is ready for tasks
# Returns: 503 Service Unavailable if at max capacity
Health Details:
curl http://localhost:8081/health
# Returns JSON with:
# - P2P connectivity status
# - DHT reachability
# - Active task count
# - Available capacity
# - Last heartbeat time
Health Criteria
Agent is healthy when:
- ✅ License valid
- ✅ P2P node connected
- ✅ At least 1 peer discovered
- ✅ Election manager running
- ✅ Task coordinator active
- ✅ HTTP API responding
Agent is ready when:
- ✅ All health checks pass
- ✅ Active tasks < max tasks
- ✅ Docker daemon reachable
- ✅ AI provider accessible
See: Health Package
Monitoring & Metrics
Prometheus Metrics
Exposed on http://localhost:8080/metrics:
Task Metrics:
chorus_tasks_active- Current active taskschorus_tasks_completed_total- Total completed taskschorus_tasks_failed_total- Total failed taskschorus_task_duration_seconds- Task execution duration histogram
P2P Metrics:
chorus_peers_connected- Number of connected peerschorus_pubsub_messages_sent_total- PubSub messages sentchorus_pubsub_messages_received_total- PubSub messages receivedchorus_dht_queries_total- DHT query countchorus_dht_cache_hits_total- DHT cache hitschorus_dht_cache_misses_total- DHT cache misses
Execution Metrics:
chorus_sandbox_containers_active- Active Docker containerschorus_sandbox_cpu_usage- Container CPU usagechorus_sandbox_memory_usage_bytes- Container memory usage
Security Metrics:
chorus_shhh_findings_total- Secrets detected by SHHHchorus_license_checks_total- License validation attemptschorus_license_failures_total- Failed license validations
See: Metrics Package
Integration Points
WHOOSH Assignment System
Agents can load dynamic assignments from WHOOSH:
# Set assignment URL
export ASSIGN_URL=https://whoosh.example.com/api/assignments/agent-123.json
# Agent fetches assignment on startup
# Assignment JSON structure:
{
"agent_id": "agent-123",
"role": "developer",
"expertise": ["rust", "go"],
"reports_to": "agent-admin",
"max_tasks": 5,
"bootstrap_peers": [
"/ip4/192.168.1.100/tcp/9000/p2p/12D3KooWABC..."
],
"join_stagger_ms": 5000
}
# Reload with SIGHUP
kill -HUP $(pidof chorus-agent)
KACHING License Server
All agents validate licenses on startup:
# License validation flow
1. Agent starts with CHORUS_LICENSE_ID
2. Connects to KACHING server (from config)
3. Validates license is:
- Valid and not expired
- Assigned to correct cluster
- Has required permissions
4. If invalid: agent exits with error
5. If valid: agent continues startup
See: Licensing
BACKBEAT Integration
Optional telemetry system for P2P operations:
export CHORUS_BACKBEAT_ENABLED=true
export CHORUS_BACKBEAT_ENDPOINT=http://backbeat.example.com
# When enabled, agent tracks:
# - P2P operation phases
# - DHT bootstrap timing
# - Election progression
# - Task execution phases
See: BACKBEAT Integration
Example Deployments
Local Development
#!/bin/bash
# Run local agent for development
export CHORUS_LICENSE_ID=dev-local-123
export CHORUS_AGENT_ID=dev-agent-1
export CHORUS_P2P_PORT=9000
export CHORUS_API_PORT=8080
export CHORUS_HEALTH_PORT=8081
export OLLAMA_ENDPOINT=http://localhost:11434
export CHORUS_DHT_ENABLED=false # Disable DHT for local dev
./chorus-agent
Docker Container
FROM debian:bookworm-slim
# Install runtime dependencies
RUN apt-get update && apt-get install -y \
ca-certificates \
docker.io \
&& rm -rf /var/lib/apt/lists/*
# Copy binary
COPY chorus-agent /usr/local/bin/chorus-agent
# Expose ports
EXPOSE 9000 8080 8081
# Run as non-root
USER nobody
ENTRYPOINT ["/usr/local/bin/chorus-agent"]
docker run -d \
--name chorus-agent-1 \
-e CHORUS_LICENSE_ID=prod-123 \
-e CHORUS_AGENT_ID=agent-1 \
-v /var/run/docker.sock:/var/run/docker.sock \
-p 9000:9000 \
-p 8080:8080 \
-p 8081:8081 \
chorus-agent:latest
Docker Swarm Service
version: "3.8"
services:
chorus-agent:
image: registry.example.com/chorus-agent:1.0.0
environment:
CHORUS_LICENSE_ID: ${CHORUS_LICENSE_ID}
CHORUS_P2P_PORT: 9000
CHORUS_API_PORT: 8080
CHORUS_DHT_ENABLED: "true"
CHORUS_BOOTSTRAP_PEERS: "/ip4/192.168.1.100/tcp/9000/p2p/12D3KooWABC..."
ASSIGN_URL: "https://whoosh.example.com/api/assignments/{{.Service.Name}}.{{.Task.Slot}}.json"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /rust/containers/WHOOSH/prompts:/prompts:ro
deploy:
replicas: 3
placement:
constraints:
- node.role == worker
networks:
- chorus-mesh
ports:
- target: 9000
published: 9000
mode: host
Troubleshooting
Agent Won't Start
Symptom: Agent exits immediately with error
Possible Causes:
-
Invalid or missing license
❌ Failed to initialize CHORUS agent: license validation failedFix: Check
CHORUS_LICENSE_IDand KACHING server connectivity -
Docker socket not accessible
❌ Failed to create P2P node: failed to create Docker clientFix: Mount
/var/run/docker.sockor check Docker daemon -
Port already in use
❌ Failed to initialize: bind: address already in useFix: Change
CHORUS_P2P_PORTor kill process on port
No Peer Discovery
Symptom: Agent starts but shows 0 connected peers
Possible Causes:
-
mDNS blocked by firewall Fix: Allow UDP port 5353, or use bootstrap peers
-
No bootstrap peers configured Fix: Set
CHORUS_BOOTSTRAP_PEERSwith valid multiaddrs -
Network isolation Fix: Ensure agents can reach each other on P2P ports
Tasks Not Executing
Symptom: Agent receives tasks but doesn't execute
Possible Causes:
-
Agent at max capacity Check:
curl localhost:8080/metrics | grep chorus_tasks_activeFix: IncreaseCHORUS_AGENT_MAX_TASKS -
Docker images not available Check:
docker images | grep chorusFix: Pull images:docker pull anthonyrawlins/chorus-rust-dev:latest -
Wrong specialization Check: Task language doesn't match agent expertise Fix: Adjust
CHORUS_AGENT_EXPERTISEor remove specialization
High Memory Usage
Symptom: Agent consuming excessive memory
Possible Causes:
-
DHT cache size too large Fix: Reduce
CHORUS_DHT_CACHE_SIZE(default 100MB) -
Too many concurrent tasks Fix: Reduce
CHORUS_AGENT_MAX_TASKS -
Memory leak in long-running containers Fix: Restart agent periodically or investigate task code
Related Documentation
- chorus-hap - Human Agent Portal binary
- chorus - Deprecated compatibility wrapper
- internal/runtime - Shared runtime initialization
- Task Execution Engine - Task execution details
- Configuration - Environment variables reference
- Deployment - Docker deployment guide
Implementation Status
| Feature | Status | Notes |
|---|---|---|
| P2P Networking | ✅ Production | libp2p, mDNS, DHT |
| Task Execution | ✅ Production | Docker sandboxing |
| License Validation | ✅ Production | KACHING integration |
| HMMM Reasoning | 🔶 Beta | Collaborative meta-discussion |
| UCXL Publishing | ✅ Production | Decision recording |
| Election | ✅ Production | Democratic leader election |
| Health Checks | ✅ Production | Liveness & readiness |
| Metrics | ✅ Production | Prometheus format |
| Assignment Loading | ✅ Production | WHOOSH integration |
| SIGHUP Reload | ✅ Production | Dynamic reconfiguration |
| BACKBEAT Telemetry | 🔶 Beta | Optional P2P tracking |
Last Updated: 2025-09-30