docs: Add Phase 3 coordination and infrastructure documentation

Comprehensive documentation for coordination, messaging, discovery, and internal systems.

Core Coordination Packages:
- pkg/election - Democratic leader election (uptime-based, heartbeat mechanism, SLURP integration)
- pkg/coordination - Meta-coordination with dependency detection (4 built-in rules)
- coordinator/ - Task orchestration and assignment (AI-powered scoring)
- discovery/ - mDNS peer discovery (automatic LAN detection)

Messaging & P2P Infrastructure:
- pubsub/ - GossipSub messaging (31 message types, role-based topics, HMMM integration)
- p2p/ - libp2p networking (DHT modes, connection management, security)

Monitoring & Health:
- pkg/metrics - Prometheus metrics (80+ metrics across 12 categories)
- pkg/health - Health monitoring (4 HTTP endpoints, enhanced checks, graceful degradation)

Internal Systems:
- internal/licensing - License validation (KACHING integration, cluster leases, fail-closed)
- internal/hapui - Human Agent Portal UI (9 commands, HMMM wizard, UCXL browser, decision voting)
- internal/backbeat - P2P operation telemetry (6 phases, beat synchronization, health reporting)

Documentation Statistics (Phase 3):
- 10 packages documented (~18,000 lines)
- 31 PubSub message types cataloged
- 80+ Prometheus metrics documented
- Complete API references with examples
- Integration patterns and best practices

Key Features Documented:
- Election: 5 triggers, candidate scoring (5 weighted components), stability windows
- Coordination: AI-powered dependency detection, cross-repo sessions, escalation handling
- PubSub: Topic patterns, message envelopes, SHHH redaction, Hypercore logging
- Metrics: All metric types with labels, Prometheus scrape config, alert rules
- Health: Liveness vs readiness, critical checks, Kubernetes integration
- Licensing: Grace periods, circuit breaker, cluster lease management
- HAP UI: Interactive terminal commands, HMMM composition wizard, web interface (beta)
- BACKBEAT: 6-phase operation tracking, beat budget estimation, drift detection

Implementation Status Marked:
- ✅ Production: Election, metrics, health, licensing, pubsub, p2p, discovery, coordinator
- 🔶 Beta: HAP web interface, BACKBEAT telemetry, advanced coordination
- 🔷 Alpha: SLURP election scoring
- ⚠️ Experimental: Meta-coordination, AI-powered dependency detection

Progress: 22/62 files complete (35%)

Next Phase: AI providers, SLURP system, API layer, reasoning engine

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

docs/comprehensive/packages/coordination.md

@@ -0,0 +1,949 @@
+# Package: pkg/coordination
+
+**Location**: `/home/tony/chorus/project-queues/active/CHORUS/pkg/coordination/`
+
+## Overview
+
+The `pkg/coordination` package provides **advanced cross-repository coordination primitives** for managing complex task dependencies and multi-agent collaboration in CHORUS. It includes AI-powered dependency detection, meta-coordination sessions, and automated escalation handling to enable sophisticated distributed development workflows.
+
+## Architecture
+
+### Coordination Layers
+
+```
+┌─────────────────────────────────────────────────┐
+│         MetaCoordinator                         │
+│   - Session management                          │
+│   - AI-powered coordination planning            │
+│   - Escalation handling                         │
+│   - SLURP integration                           │
+└─────────────────┬───────────────────────────────┘
+                  │
+┌─────────────────▼───────────────────────────────┐
+│         DependencyDetector                      │
+│   - Cross-repo dependency detection             │
+│   - Rule-based pattern matching                 │
+│   - Relationship analysis                       │
+└─────────────────┬───────────────────────────────┘
+                  │
+┌─────────────────▼───────────────────────────────┐
+│         PubSub (HMMM Meta-Discussion)           │
+│   - Coordination messages                       │
+│   - Session broadcasts                          │
+│   - Escalation notifications                    │
+└─────────────────────────────────────────────────┘
+```
+
+## Core Components
+
+### MetaCoordinator
+
+Manages advanced cross-repository coordination and multi-agent collaboration sessions.
+
+```go
+type MetaCoordinator struct {
+    pubsub               *pubsub.PubSub
+    ctx                  context.Context
+    dependencyDetector   *DependencyDetector
+    slurpIntegrator      *integration.SlurpEventIntegrator
+
+    // Active coordination sessions
+    activeSessions       map[string]*CoordinationSession
+    sessionLock          sync.RWMutex
+
+    // Configuration
+    maxSessionDuration   time.Duration    // Default: 30 minutes
+    maxParticipants      int             // Default: 5
+    escalationThreshold  int             // Default: 10 messages
+}
+```
+
+**Key Responsibilities:**
+- Create and manage coordination sessions
+- Generate AI-powered coordination plans
+- Monitor session progress and health
+- Escalate to humans when needed
+- Generate SLURP events from coordination outcomes
+- Integrate with HMMM for meta-discussion
+
+### DependencyDetector
+
+Analyzes tasks across repositories to detect relationships and dependencies.
+
+```go
+type DependencyDetector struct {
+    pubsub            *pubsub.PubSub
+    ctx               context.Context
+    knownTasks        map[string]*TaskContext
+    dependencyRules   []DependencyRule
+    coordinationHops  int  // Default: 3
+}
+```
+
+**Key Responsibilities:**
+- Track tasks across multiple repositories
+- Apply pattern-based dependency detection rules
+- Identify task relationships (API contracts, schema changes, etc.)
+- Broadcast dependency alerts
+- Trigger coordination sessions
+
+### CoordinationSession
+
+Represents an active multi-agent coordination session.
+
+```go
+type CoordinationSession struct {
+    SessionID           string
+    Type                string  // dependency, conflict, planning
+    Participants        map[string]*Participant
+    TasksInvolved       []*TaskContext
+    Messages            []CoordinationMessage
+    Status              string  // active, resolved, escalated
+    CreatedAt           time.Time
+    LastActivity        time.Time
+    Resolution          string
+    EscalationReason    string
+}
+```
+
+**Session Types:**
+- **dependency**: Coordinating dependent tasks across repos
+- **conflict**: Resolving conflicts or competing changes
+- **planning**: Joint planning for complex multi-repo features
+
+**Session States:**
+- **active**: Session in progress
+- **resolved**: Consensus reached, coordination complete
+- **escalated**: Requires human intervention
+
+## Data Structures
+
+### TaskContext
+
+Represents a task with its repository and project context for dependency analysis.
+
+```go
+type TaskContext struct {
+    TaskID      int
+    ProjectID   int
+    Repository  string
+    Title       string
+    Description string
+    Keywords    []string
+    AgentID     string
+    ClaimedAt   time.Time
+}
+```
+
+### Participant
+
+Represents an agent participating in a coordination session.
+
+```go
+type Participant struct {
+    AgentID      string
+    PeerID       string
+    Repository   string
+    Capabilities []string
+    LastSeen     time.Time
+    Active       bool
+}
+```
+
+### CoordinationMessage
+
+A message within a coordination session.
+
+```go
+type CoordinationMessage struct {
+    MessageID   string
+    FromAgentID string
+    FromPeerID  string
+    Content     string
+    MessageType string  // proposal, question, agreement, concern
+    Timestamp   time.Time
+    Metadata    map[string]interface{}
+}
+```
+
+**Message Types:**
+- **proposal**: Proposed solution or approach
+- **question**: Request for clarification
+- **agreement**: Agreement with proposal
+- **concern**: Concern or objection
+
+### TaskDependency
+
+Represents a detected relationship between tasks.
+
+```go
+type TaskDependency struct {
+    Task1       *TaskContext
+    Task2       *TaskContext
+    Relationship string   // Rule name (e.g., "API_Contract")
+    Confidence   float64  // 0.0 - 1.0
+    Reason       string   // Human-readable explanation
+    DetectedAt   time.Time
+}
+```
+
+### DependencyRule
+
+Defines how to detect task relationships.
+
+```go
+type DependencyRule struct {
+    Name        string
+    Description string
+    Keywords    []string
+    Validator   func(task1, task2 *TaskContext) (bool, string)
+}
+```
+
+## Dependency Detection
+
+### Built-in Detection Rules
+
+#### 1. API Contract Rule
+
+Detects dependencies between API definitions and implementations.
+
+```go
+{
+    Name:        "API_Contract",
+    Description: "Tasks involving API contracts and implementations",
+    Keywords:    []string{"api", "endpoint", "contract", "interface", "schema"},
+    Validator: func(task1, task2 *TaskContext) (bool, string) {
+        text1 := strings.ToLower(task1.Title + " " + task1.Description)
+        text2 := strings.ToLower(task2.Title + " " + task2.Description)
+
+        if (strings.Contains(text1, "api") && strings.Contains(text2, "implement")) ||
+           (strings.Contains(text2, "api") && strings.Contains(text1, "implement")) {
+            return true, "API definition and implementation dependency"
+        }
+        return false, ""
+    },
+}
+```
+
+**Example Detection:**
+- Task 1: "Define user authentication API"
+- Task 2: "Implement authentication endpoint"
+- **Detected**: API_Contract dependency
+
+#### 2. Database Schema Rule
+
+Detects schema changes affecting multiple services.
+
+```go
+{
+    Name:        "Database_Schema",
+    Description: "Database schema changes affecting multiple services",
+    Keywords:    []string{"database", "schema", "migration", "table", "model"},
+    Validator: func(task1, task2 *TaskContext) (bool, string) {
+        // Checks for database-related keywords in both tasks
+        // Returns true if both tasks involve database work
+    },
+}
+```
+
+**Example Detection:**
+- Task 1: "Add user preferences table"
+- Task 2: "Update user service for preferences"
+- **Detected**: Database_Schema dependency
+
+#### 3. Configuration Dependency Rule
+
+Detects configuration changes affecting multiple components.
+
+```go
+{
+    Name:        "Configuration_Dependency",
+    Description: "Configuration changes affecting multiple components",
+    Keywords:    []string{"config", "environment", "settings", "parameters"},
+}
+```
+
+**Example Detection:**
+- Task 1: "Add feature flag for new UI"
+- Task 2: "Implement feature flag checks in backend"
+- **Detected**: Configuration_Dependency
+
+#### 4. Security Compliance Rule
+
+Detects security changes requiring coordinated implementation.
+
+```go
+{
+    Name:        "Security_Compliance",
+    Description: "Security changes requiring coordinated implementation",
+    Keywords:    []string{"security", "auth", "permission", "token", "encrypt"},
+}
+```
+
+**Example Detection:**
+- Task 1: "Implement JWT token refresh"
+- Task 2: "Update authentication middleware"
+- **Detected**: Security_Compliance dependency
+
+### Custom Rules
+
+Add project-specific dependency detection:
+
+```go
+customRule := DependencyRule{
+    Name:        "GraphQL_Schema",
+    Description: "GraphQL schema and resolver dependencies",
+    Keywords:    []string{"graphql", "schema", "resolver", "query", "mutation"},
+    Validator: func(task1, task2 *TaskContext) (bool, string) {
+        text1 := strings.ToLower(task1.Title + " " + task1.Description)
+        text2 := strings.ToLower(task2.Title + " " + task2.Description)
+
+        hasSchema := strings.Contains(text1, "schema") || strings.Contains(text2, "schema")
+        hasResolver := strings.Contains(text1, "resolver") || strings.Contains(text2, "resolver")
+
+        if hasSchema && hasResolver {
+            return true, "GraphQL schema and resolver must be coordinated"
+        }
+        return false, ""
+    },
+}
+
+dependencyDetector.AddCustomRule(customRule)
+```
+
+## Coordination Flow
+
+### 1. Task Registration and Detection
+
+```
+Task Claimed by Agent A → RegisterTask() → DependencyDetector
+                              ↓
+                        detectDependencies()
+                              ↓
+            Apply all dependency rules to known tasks
+                              ↓
+                   Dependency detected? → Yes → announceDependency()
+                              ↓                           ↓
+                             No                  MetaCoordinator
+```
+
+### 2. Dependency Announcement
+
+```go
+// Dependency detector announces to HMMM meta-discussion
+coordMsg := map[string]interface{}{
+    "message_type":   "dependency_detected",
+    "dependency":     dep,
+    "coordination_request": "Cross-repository dependency detected...",
+    "agents_involved": [agentA, agentB],
+    "repositories":    [repoA, repoB],
+    "hop_count":       0,
+    "max_hops":        3,
+}
+
+pubsub.PublishHmmmMessage(MetaDiscussion, coordMsg)
+```
+
+### 3. Session Creation
+
+```
+MetaCoordinator receives dependency_detected message
+                ↓
+    handleDependencyDetection()
+                ↓
+    Create CoordinationSession
+                ↓
+    Add participating agents
+                ↓
+    Generate AI coordination plan
+                ↓
+    Broadcast plan to participants
+```
+
+### 4. AI-Powered Coordination Planning
+
+```go
+prompt := `
+You are an expert AI project coordinator managing a distributed development team.
+
+SITUATION:
+- A dependency has been detected between two tasks in different repositories
+- Task 1: repo1/title #42 (Agent: agent-001)
+- Task 2: repo2/title #43 (Agent: agent-002)
+- Relationship: API_Contract
+- Reason: API definition and implementation dependency
+
+COORDINATION REQUIRED:
+Generate a concise coordination plan that addresses:
+1. What specific coordination is needed between the agents
+2. What order should tasks be completed in (if any)
+3. What information/artifacts need to be shared
+4. What potential conflicts to watch for
+5. Success criteria for coordinated completion
+`
+
+plan := reasoning.GenerateResponse(ctx, "phi3", prompt)
+```
+
+**Plan Output Example:**
+```
+COORDINATION PLAN:
+
+1. SEQUENCE:
+   - Task 1 (API definition) must be completed first
+   - Task 2 (implementation) depends on finalized API contract
+
+2. INFORMATION SHARING:
+   - Agent-001 must share: API specification document, endpoint definitions
+   - Agent-002 must share: Implementation plan, integration tests
+
+3. COORDINATION POINTS:
+   - Review API spec before implementation begins
+   - Daily sync on implementation progress
+   - Joint testing before completion
+
+4. POTENTIAL CONFLICTS:
+   - API spec changes during implementation
+   - Performance requirements not captured in spec
+   - Authentication/authorization approach
+
+5. SUCCESS CRITERIA:
+   - API spec reviewed and approved
+   - Implementation matches spec
+   - Integration tests pass
+   - Documentation complete
+```
+
+### 5. Session Progress Monitoring
+
+```
+Agents respond to coordination plan
+                ↓
+    handleCoordinationResponse()
+                ↓
+    Add message to session
+                ↓
+    Update participant activity
+                ↓
+    evaluateSessionProgress()
+                ↓
+    ┌──────────────────────┐
+    │ Check conditions:    │
+    │ - Message count      │
+    │ - Session duration   │
+    │ - Agreement keywords │
+    └──────┬───────────────┘
+           │
+    ┌──────▼──────┬──────────────┐
+    │             │              │
+Consensus?    Too long?    Too many msgs?
+    │             │              │
+Resolved      Escalate      Escalate
+```
+
+### 6. Session Resolution
+
+**Consensus Reached:**
+```go
+// Detect agreement in recent messages
+agreementKeywords := []string{
+    "agree", "sounds good", "approved", "looks good", "confirmed"
+}
+
+if agreementCount >= len(participants)-1 {
+    resolveSession(session, "Consensus reached among participants")
+}
+```
+
+**Session Resolved:**
+1. Update session status to "resolved"
+2. Record resolution reason
+3. Generate SLURP event (if integrator available)
+4. Broadcast resolution to participants
+5. Clean up after timeout
+
+### 7. Session Escalation
+
+**Escalation Triggers:**
+- Message count exceeds threshold (default: 10)
+- Session duration exceeds limit (default: 30 minutes)
+- Explicit escalation request from agent
+
+**Escalation Process:**
+```go
+escalateSession(session, reason)
+    ↓
+Update status to "escalated"
+    ↓
+Generate SLURP event for human review
+    ↓
+Broadcast escalation notification
+    ↓
+Human intervention required
+```
+
+## SLURP Integration
+
+### Event Generation from Sessions
+
+When sessions are resolved or escalated, the MetaCoordinator generates SLURP events:
+
+```go
+discussionContext := integration.HmmmDiscussionContext{
+    DiscussionID:      session.SessionID,
+    SessionID:         session.SessionID,
+    Participants:      [agentIDs],
+    StartTime:         session.CreatedAt,
+    EndTime:           session.LastActivity,
+    Messages:          hmmmMessages,
+    ConsensusReached:  (outcome == "resolved"),
+    ConsensusStrength: 0.9,  // 0.3 for escalated, 0.5 for other
+    OutcomeType:       outcome,  // "resolved" or "escalated"
+    ProjectPath:       projectPath,
+    RelatedTasks:      [taskIDs],
+    Metadata: {
+        "session_type":        session.Type,
+        "session_status":      session.Status,
+        "resolution":          session.Resolution,
+        "escalation_reason":   session.EscalationReason,
+        "message_count":       len(session.Messages),
+        "participant_count":   len(session.Participants),
+    },
+}
+
+slurpIntegrator.ProcessHmmmDiscussion(ctx, discussionContext)
+```
+
+**SLURP Event Outcomes:**
+- **Resolved sessions**: High consensus (0.9), successful coordination
+- **Escalated sessions**: Low consensus (0.3), human intervention needed
+- **Other outcomes**: Medium consensus (0.5)
+
+### Policy Learning
+
+SLURP uses coordination session data to learn:
+- Effective coordination patterns
+- Common dependency types
+- Escalation triggers
+- Agent collaboration efficiency
+- Task complexity indicators
+
+## PubSub Message Types
+
+### 1. dependency_detected
+
+Announces a detected dependency between tasks.
+
+```json
+{
+  "message_type": "dependency_detected",
+  "dependency": {
+    "task1": {
+      "task_id": 42,
+      "project_id": 1,
+      "repository": "backend-api",
+      "title": "Define user authentication API",
+      "agent_id": "agent-001"
+    },
+    "task2": {
+      "task_id": 43,
+      "project_id": 2,
+      "repository": "frontend-app",
+      "title": "Implement login page",
+      "agent_id": "agent-002"
+    },
+    "relationship": "API_Contract",
+    "confidence": 0.8,
+    "reason": "API definition and implementation dependency",
+    "detected_at": "2025-09-30T10:00:00Z"
+  },
+  "coordination_request": "Cross-repository dependency detected...",
+  "agents_involved": ["agent-001", "agent-002"],
+  "repositories": ["backend-api", "frontend-app"],
+  "hop_count": 0,
+  "max_hops": 3
+}
+```
+
+### 2. coordination_plan
+
+Broadcasts AI-generated coordination plan to participants.
+
+```json
+{
+  "message_type": "coordination_plan",
+  "session_id": "dep_1_42_1727692800",
+  "plan": "COORDINATION PLAN:\n1. SEQUENCE:\n...",
+  "tasks_involved": [taskContext1, taskContext2],
+  "participants": {
+    "agent-001": { "agent_id": "agent-001", "repository": "backend-api" },
+    "agent-002": { "agent_id": "agent-002", "repository": "frontend-app" }
+  },
+  "message": "Coordination plan generated for dependency: API_Contract"
+}
+```
+
+### 3. coordination_response
+
+Agent response to coordination plan or session message.
+
+```json
+{
+  "message_type": "coordination_response",
+  "session_id": "dep_1_42_1727692800",
+  "agent_id": "agent-001",
+  "response": "I agree with the proposed sequence. API spec will be ready by EOD.",
+  "timestamp": "2025-09-30T10:05:00Z"
+}
+```
+
+### 4. session_message
+
+General message within a coordination session.
+
+```json
+{
+  "message_type": "session_message",
+  "session_id": "dep_1_42_1727692800",
+  "from_agent": "agent-002",
+  "content": "Can we schedule a quick sync to review the API spec?",
+  "timestamp": "2025-09-30T10:10:00Z"
+}
+```
+
+### 5. escalation
+
+Session escalated to human intervention.
+
+```json
+{
+  "message_type": "escalation",
+  "session_id": "dep_1_42_1727692800",
+  "escalation_reason": "Message limit exceeded - human intervention needed",
+  "session_summary": "Session dep_1_42_1727692800 (dependency): 2 participants, 12 messages, duration 35m",
+  "participants": { /* participant info */ },
+  "tasks_involved": [ /* task contexts */ ],
+  "requires_human": true
+}
+```
+
+### 6. resolution
+
+Session successfully resolved.
+
+```json
+{
+  "message_type": "resolution",
+  "session_id": "dep_1_42_1727692800",
+  "resolution": "Consensus reached among participants",
+  "summary": "Session dep_1_42_1727692800 (dependency): 2 participants, 8 messages, duration 15m"
+}
+```
+
+## Usage Examples
+
+### Basic Setup
+
+```go
+import (
+    "context"
+    "chorus/pkg/coordination"
+    "chorus/pubsub"
+)
+
+// Create MetaCoordinator
+mc := coordination.NewMetaCoordinator(ctx, pubsubInstance)
+
+// Optionally attach SLURP integrator
+mc.SetSlurpIntegrator(slurpIntegrator)
+
+// MetaCoordinator automatically:
+// - Initializes DependencyDetector
+// - Sets up HMMM message handlers
+// - Starts session cleanup loop
+```
+
+### Register Tasks for Dependency Detection
+
+```go
+// Agent claims a task
+taskContext := &coordination.TaskContext{
+    TaskID:      42,
+    ProjectID:   1,
+    Repository:  "backend-api",
+    Title:       "Define user authentication API",
+    Description: "Create OpenAPI spec for user auth endpoints",
+    Keywords:    []string{"api", "authentication", "openapi"},
+    AgentID:     "agent-001",
+    ClaimedAt:   time.Now(),
+}
+
+mc.dependencyDetector.RegisterTask(taskContext)
+```
+
+### Add Custom Dependency Rule
+
+```go
+// Add project-specific rule
+microserviceRule := coordination.DependencyRule{
+    Name:        "Microservice_Interface",
+    Description: "Microservice interface and consumer dependencies",
+    Keywords:    []string{"microservice", "interface", "consumer", "producer"},
+    Validator: func(task1, task2 *coordination.TaskContext) (bool, string) {
+        t1 := strings.ToLower(task1.Title + " " + task1.Description)
+        t2 := strings.ToLower(task2.Title + " " + task2.Description)
+
+        hasProducer := strings.Contains(t1, "producer") || strings.Contains(t2, "producer")
+        hasConsumer := strings.Contains(t1, "consumer") || strings.Contains(t2, "consumer")
+
+        if hasProducer && hasConsumer {
+            return true, "Microservice producer and consumer must coordinate"
+        }
+        return false, ""
+    },
+}
+
+mc.dependencyDetector.AddCustomRule(microserviceRule)
+```
+
+### Query Active Sessions
+
+```go
+// Get all active coordination sessions
+sessions := mc.GetActiveSessions()
+
+for sessionID, session := range sessions {
+    fmt.Printf("Session %s:\n", sessionID)
+    fmt.Printf("  Type: %s\n", session.Type)
+    fmt.Printf("  Status: %s\n", session.Status)
+    fmt.Printf("  Participants: %d\n", len(session.Participants))
+    fmt.Printf("  Messages: %d\n", len(session.Messages))
+    fmt.Printf("  Duration: %v\n", time.Since(session.CreatedAt))
+}
+```
+
+### Monitor Coordination Events
+
+```go
+// Set custom HMMM message handler
+pubsub.SetHmmmMessageHandler(func(msg pubsub.Message, from peer.ID) {
+    switch msg.Data["message_type"] {
+    case "dependency_detected":
+        fmt.Printf("🔗 Dependency detected: %v\n", msg.Data)
+    case "coordination_plan":
+        fmt.Printf("📋 Coordination plan: %v\n", msg.Data)
+    case "escalation":
+        fmt.Printf("🚨 Escalation: %v\n", msg.Data)
+    case "resolution":
+        fmt.Printf("✅ Resolution: %v\n", msg.Data)
+    }
+})
+```
+
+## Configuration
+
+### MetaCoordinator Configuration
+
+```go
+mc := coordination.NewMetaCoordinator(ctx, ps)
+
+// Adjust session parameters
+mc.maxSessionDuration = 45 * time.Minute   // Extend session timeout
+mc.maxParticipants = 10                     // Support larger teams
+mc.escalationThreshold = 15                 // More messages before escalation
+```
+
+### DependencyDetector Configuration
+
+```go
+dd := mc.dependencyDetector
+
+// Adjust coordination hop limit
+dd.coordinationHops = 5  // Allow deeper meta-discussion chains
+```
+
+## Session Lifecycle Management
+
+### Automatic Cleanup
+
+Sessions are automatically cleaned up by the session cleanup loop:
+
+```go
+// Runs every 10 minutes
+func (mc *MetaCoordinator) cleanupInactiveSessions() {
+    for sessionID, session := range mc.activeSessions {
+        // Remove sessions older than 2 hours OR already resolved/escalated
+        if time.Since(session.LastActivity) > 2*time.Hour ||
+           session.Status == "resolved" ||
+           session.Status == "escalated" {
+            delete(mc.activeSessions, sessionID)
+        }
+    }
+}
+```
+
+**Cleanup Criteria:**
+- Session inactive for 2+ hours
+- Session status is "resolved"
+- Session status is "escalated"
+
+### Manual Session Management
+
+```go
+// Not exposed in current API, but could be added:
+
+// Force resolve session
+mc.resolveSession(session, "Manual resolution by admin")
+
+// Force escalate session
+mc.escalateSession(session, "Manual escalation requested")
+
+// Cancel/close session
+mc.closeSession(sessionID)
+```
+
+## Performance Considerations
+
+### Memory Usage
+
+- **TaskContext Storage**: ~500 bytes per task
+- **Active Sessions**: ~5KB per session (varies with message count)
+- **Dependency Rules**: ~1KB per rule
+
+**Typical Usage**: 100 tasks + 10 sessions = ~100KB
+
+### CPU Usage
+
+- **Dependency Detection**: O(N²) where N = number of tasks per repository
+- **Rule Evaluation**: O(R) where R = number of rules
+- **Session Monitoring**: Periodic evaluation (every message received)
+
+**Optimization**: Dependency detection skips same-repository comparisons.
+
+### Network Usage
+
+- **Dependency Announcements**: ~2KB per dependency
+- **Coordination Plans**: ~5KB per plan (includes full context)
+- **Session Messages**: ~1KB per message
+- **SLURP Events**: ~10KB per event (includes full session history)
+
+## Best Practices
+
+### 1. Rule Design
+
+**Good Rule:**
+```go
+// Specific, actionable, clear success criteria
+{
+    Name: "Database_Migration",
+    Keywords: []string{"migration", "schema", "database"},
+    Validator: func(t1, t2 *TaskContext) (bool, string) {
+        // Clear matching logic
+        // Specific reason returned
+    },
+}
+```
+
+**Bad Rule:**
+```go
+// Too broad, unclear coordination needed
+{
+    Name: "Backend_Tasks",
+    Keywords: []string{"backend"},
+    Validator: func(t1, t2 *TaskContext) (bool, string) {
+        return strings.Contains(t1.Title, "backend") &&
+               strings.Contains(t2.Title, "backend"), "Both backend tasks"
+    },
+}
+```
+
+### 2. Session Participation
+
+- **Respond promptly**: Keep sessions moving
+- **Be explicit**: Use clear agreement/disagreement language
+- **Stay focused**: Don't derail session with unrelated topics
+- **Escalate when stuck**: Don't let sessions drag on indefinitely
+
+### 3. AI Plan Quality
+
+AI plans are most effective when:
+- Task descriptions are detailed
+- Dependencies are clear
+- Agent capabilities are well-defined
+- Historical context is available
+
+### 4. SLURP Integration
+
+For best SLURP learning:
+- Enable SLURP integrator at startup
+- Ensure all sessions generate events (resolved or escalated)
+- Provide rich task metadata
+- Include project context in task descriptions
+
+## Troubleshooting
+
+### Dependencies Not Detected
+
+**Symptoms**: Related tasks not triggering coordination.
+
+**Checks:**
+1. Verify tasks registered with detector: `dd.GetKnownTasks()`
+2. Check rule keywords match task content
+3. Test validator logic with task pairs
+4. Verify tasks are from different repositories
+5. Check PubSub connection for announcements
+
+### Sessions Not Escalating
+
+**Symptoms**: Long-running sessions without escalation.
+
+**Checks:**
+1. Verify escalation threshold: `mc.escalationThreshold`
+2. Check session duration limit: `mc.maxSessionDuration`
+3. Verify message count in session
+4. Check for agreement keywords in messages
+5. Test escalation logic manually
+
+### AI Plans Not Generated
+
+**Symptoms**: Sessions created but no coordination plan.
+
+**Checks:**
+1. Verify reasoning engine available: `reasoning.GenerateResponse()`
+2. Check AI model configuration
+3. Verify network connectivity to AI provider
+4. Check reasoning engine error logs
+5. Test with simpler dependency
+
+### SLURP Events Not Generated
+
+**Symptoms**: Sessions complete but no SLURP events.
+
+**Checks:**
+1. Verify SLURP integrator attached: `mc.SetSlurpIntegrator()`
+2. Check SLURP integrator initialization
+3. Verify session outcome triggers event generation
+4. Check SLURP integrator error logs
+5. Test event generation manually
+
+## Future Enhancements
+
+### Planned Features
+
+1. **Machine Learning Rules**: Learn dependency patterns from historical data
+2. **Automated Testing**: Generate integration tests for coordinated tasks
+3. **Visualization**: Web UI for monitoring active sessions
+4. **Advanced Metrics**: Track coordination efficiency and success rates
+5. **Multi-Repo CI/CD**: Coordinate deployments across dependent services
+6. **Conflict Resolution**: AI-powered conflict resolution suggestions
+7. **Predictive Coordination**: Predict dependencies before tasks are claimed
+
+## See Also
+
+- [coordinator/](coordinator.md) - Task coordinator integration
+- [pubsub/](../pubsub.md) - PubSub messaging for coordination
+- [pkg/integration/](integration.md) - SLURP integration
+- [pkg/hmmm/](hmmm.md) - HMMM meta-discussion system
+- [reasoning/](../reasoning.md) - AI reasoning engine for planning
+- [internal/logging/](../internal/logging.md) - Hypercore logging

docs/comprehensive/packages/coordinator.md

@@ -0,0 +1,750 @@
+# Package: coordinator
+
+**Location**: `/home/tony/chorus/project-queues/active/CHORUS/coordinator/`
+
+## Overview
+
+The `coordinator` package provides the **TaskCoordinator** - the main orchestrator for distributed task management in CHORUS. It handles task discovery, intelligent assignment, execution coordination, and real-time progress tracking across multiple repositories and agents. The coordinator integrates with the PubSub system for role-based collaboration and uses AI-powered execution engines for autonomous task completion.
+
+## Core Components
+
+### TaskCoordinator
+
+The central orchestrator managing task lifecycle across the distributed CHORUS network.
+
+```go
+type TaskCoordinator struct {
+    pubsub     *pubsub.PubSub
+    hlog       *logging.HypercoreLog
+    ctx        context.Context
+    config     *config.Config
+    hmmmRouter *hmmm.Router
+
+    // Repository management
+    providers    map[int]repository.TaskProvider // projectID -> provider
+    providerLock sync.RWMutex
+    factory      repository.ProviderFactory
+
+    // Task management
+    activeTasks map[string]*ActiveTask // taskKey -> active task
+    taskLock    sync.RWMutex
+    taskMatcher repository.TaskMatcher
+    taskTracker TaskProgressTracker
+
+    // Task execution
+    executionEngine execution.TaskExecutionEngine
+
+    // Agent tracking
+    nodeID    string
+    agentInfo *repository.AgentInfo
+
+    // Sync settings
+    syncInterval time.Duration
+    lastSync     map[int]time.Time
+    syncLock     sync.RWMutex
+}
+```
+
+**Key Responsibilities:**
+- Discover available tasks across multiple repositories
+- Score and assign tasks based on agent capabilities and expertise
+- Coordinate task execution with AI-powered execution engines
+- Track active tasks and broadcast progress updates
+- Request and coordinate multi-agent collaboration
+- Integrate with HMMM for meta-discussion and coordination
+
+### ActiveTask
+
+Represents a task currently being worked on by an agent.
+
+```go
+type ActiveTask struct {
+    Task      *repository.Task
+    Provider  repository.TaskProvider
+    ProjectID int
+    ClaimedAt time.Time
+    Status    string // claimed, working, completed, failed
+    AgentID   string
+    Results   map[string]interface{}
+}
+```
+
+**Task Lifecycle States:**
+1. **claimed** - Task has been claimed by an agent
+2. **working** - Agent is actively executing the task
+3. **completed** - Task finished successfully
+4. **failed** - Task execution failed
+
+### TaskProgressTracker Interface
+
+Callback interface for tracking task progress and updating availability broadcasts.
+
+```go
+type TaskProgressTracker interface {
+    AddTask(taskID string)
+    RemoveTask(taskID string)
+}
+```
+
+This interface ensures availability broadcasts accurately reflect current workload.
+
+## Task Coordination Flow
+
+### 1. Initialization
+
+```go
+coordinator := NewTaskCoordinator(
+    ctx,
+    ps,           // PubSub instance
+    hlog,         // Hypercore log
+    cfg,          // Agent configuration
+    nodeID,       // P2P node ID
+    hmmmRouter,   // HMMM router for meta-discussion
+    tracker,      // Task progress tracker
+)
+
+coordinator.Start()
+```
+
+**Initialization Process:**
+1. Creates agent info from configuration
+2. Sets up task execution engine with AI providers
+3. Announces agent role and capabilities via PubSub
+4. Starts task discovery loop
+5. Begins listening for role-based messages
+
+### 2. Task Discovery and Assignment
+
+**Discovery Loop** (runs every 30 seconds):
+```
+taskDiscoveryLoop() ->
+  (Discovery now handled by WHOOSH integration)
+```
+
+**Task Evaluation** (`shouldProcessTask`):
+```go
+func (tc *TaskCoordinator) shouldProcessTask(task *repository.Task) bool {
+    // 1. Check capacity: currentTasks < maxTasks
+    // 2. Check if already assigned to this agent
+    // 3. Score task fit for agent capabilities
+    // 4. Return true if score > 0.5 threshold
+}
+```
+
+**Task Scoring:**
+- Agent role matches required role
+- Agent expertise matches required expertise
+- Current workload vs capacity
+- Task priority level
+- Historical performance scores
+
+### 3. Task Claiming and Processing
+
+```
+processTask() flow:
+  1. Evaluate if collaboration needed (shouldRequestCollaboration)
+  2. Request collaboration via PubSub if needed
+  3. Claim task through repository provider
+  4. Create ActiveTask and store in activeTasks map
+  5. Log claim to Hypercore
+  6. Announce claim via PubSub (TaskProgress message)
+  7. Seed HMMM meta-discussion room for task
+  8. Start execution in background goroutine
+```
+
+**Collaboration Request Criteria:**
+- Task priority >= 8 (high priority)
+- Task requires expertise agent doesn't have
+- Complex multi-component tasks
+
+### 4. Task Execution
+
+**AI-Powered Execution** (`executeTaskWithAI`):
+
+```go
+executionRequest := &execution.TaskExecutionRequest{
+    ID:          "repo:taskNumber",
+    Type:        determineTaskType(task), // bug_fix, feature_development, etc.
+    Description: buildTaskDescription(task),
+    Context:     buildTaskContext(task),
+    Requirements: &execution.TaskRequirements{
+        AIModel:        "", // Auto-selected based on role
+        SandboxType:    "docker",
+        RequiredTools:  []string{"git", "curl"},
+        EnvironmentVars: map[string]string{
+            "TASK_ID":    taskID,
+            "REPOSITORY": repoName,
+            "AGENT_ID":   agentID,
+            "AGENT_ROLE": agentRole,
+        },
+    },
+    Timeout: 10 * time.Minute,
+}
+
+result := tc.executionEngine.ExecuteTask(ctx, executionRequest)
+```
+
+**Task Type Detection:**
+- **bug_fix** - Keywords: "bug", "fix"
+- **feature_development** - Keywords: "feature", "implement"
+- **testing** - Keywords: "test"
+- **documentation** - Keywords: "doc", "documentation"
+- **refactoring** - Keywords: "refactor"
+- **code_review** - Keywords: "review"
+- **development** - Default for general tasks
+
+**Fallback Mock Execution:**
+If AI execution engine is unavailable or fails, falls back to mock execution with simulated work time.
+
+### 5. Task Completion
+
+```
+executeTask() completion flow:
+  1. Update ActiveTask status to "completed"
+  2. Complete task through repository provider
+  3. Remove from activeTasks map
+  4. Update TaskProgressTracker
+  5. Log completion to Hypercore
+  6. Announce completion via PubSub
+```
+
+**Task Result Structure:**
+```go
+type TaskResult struct {
+    Success  bool
+    Message  string
+    Metadata map[string]interface{} // Includes:
+                                     // - execution_type (ai_powered/mock)
+                                     // - duration
+                                     // - commands_executed
+                                     // - files_generated
+                                     // - resource_usage
+                                     // - artifacts
+}
+```
+
+## PubSub Integration
+
+### Published Message Types
+
+#### 1. RoleAnnouncement
+**Topic**: `hmmm/meta-discussion/v1`
+**Frequency**: Once on startup, when capabilities change
+
+```json
+{
+  "type": "role_announcement",
+  "from": "peer_id",
+  "from_role": "Senior Backend Developer",
+  "data": {
+    "agent_id": "agent-001",
+    "node_id": "Qm...",
+    "role": "Senior Backend Developer",
+    "expertise": ["Go", "PostgreSQL", "Kubernetes"],
+    "capabilities": ["code", "test", "deploy"],
+    "max_tasks": 3,
+    "current_tasks": 0,
+    "status": "ready",
+    "specialization": "microservices"
+  }
+}
+```
+
+#### 2. TaskProgress
+**Topic**: `CHORUS/coordination/v1`
+**Frequency**: On claim, start, completion
+
+**Task Claim:**
+```json
+{
+  "type": "task_progress",
+  "from": "peer_id",
+  "from_role": "Senior Backend Developer",
+  "thread_id": "task-myrepo-42",
+  "data": {
+    "task_number": 42,
+    "repository": "myrepo",
+    "title": "Add authentication endpoint",
+    "agent_id": "agent-001",
+    "agent_role": "Senior Backend Developer",
+    "claim_time": "2025-09-30T10:00:00Z",
+    "estimated_completion": "2025-09-30T11:00:00Z"
+  }
+}
+```
+
+**Task Status Update:**
+```json
+{
+  "type": "task_progress",
+  "from": "peer_id",
+  "from_role": "Senior Backend Developer",
+  "thread_id": "task-myrepo-42",
+  "data": {
+    "task_number": 42,
+    "repository": "myrepo",
+    "agent_id": "agent-001",
+    "agent_role": "Senior Backend Developer",
+    "status": "started" | "completed",
+    "timestamp": "2025-09-30T10:05:00Z"
+  }
+}
+```
+
+#### 3. TaskHelpRequest
+**Topic**: `hmmm/meta-discussion/v1`
+**Frequency**: When collaboration needed
+
+```json
+{
+  "type": "task_help_request",
+  "from": "peer_id",
+  "from_role": "Senior Backend Developer",
+  "to_roles": ["Database Specialist"],
+  "required_expertise": ["PostgreSQL", "Query Optimization"],
+  "priority": "high",
+  "thread_id": "task-myrepo-42",
+  "data": {
+    "task_number": 42,
+    "repository": "myrepo",
+    "title": "Optimize database queries",
+    "required_role": "Database Specialist",
+    "required_expertise": ["PostgreSQL", "Query Optimization"],
+    "priority": 8,
+    "requester_role": "Senior Backend Developer",
+    "reason": "expertise_gap"
+  }
+}
+```
+
+### Received Message Types
+
+#### 1. TaskHelpRequest
+**Handler**: `handleTaskHelpRequest`
+
+**Response Logic:**
+1. Check if agent has required expertise
+2. Verify agent has available capacity (currentTasks < maxTasks)
+3. If can help, send TaskHelpResponse
+4. Reflect offer into HMMM per-issue room
+
+**Response Message:**
+```json
+{
+  "type": "task_help_response",
+  "from": "peer_id",
+  "from_role": "Database Specialist",
+  "thread_id": "task-myrepo-42",
+  "data": {
+    "agent_id": "agent-002",
+    "agent_role": "Database Specialist",
+    "expertise": ["PostgreSQL", "Query Optimization", "Indexing"],
+    "availability": 2,
+    "offer_type": "collaboration",
+    "response_to": { /* original help request data */ }
+  }
+}
+```
+
+#### 2. ExpertiseRequest
+**Handler**: `handleExpertiseRequest`
+
+Processes requests for specific expertise areas.
+
+#### 3. CoordinationRequest
+**Handler**: `handleCoordinationRequest`
+
+Handles coordination requests for multi-agent tasks.
+
+#### 4. RoleAnnouncement
+**Handler**: `handleRoleAnnouncement`
+
+Logs when other agents announce their roles and capabilities.
+
+## HMMM Integration
+
+### Per-Issue Room Seeding
+
+When a task is claimed, the coordinator seeds a HMMM meta-discussion room:
+
+```go
+seedMsg := hmmm.Message{
+    Version:   1,
+    Type:      "meta_msg",
+    IssueID:   int64(taskNumber),
+    ThreadID:  fmt.Sprintf("issue-%d", taskNumber),
+    MsgID:     uuid.New().String(),
+    NodeID:    nodeID,
+    HopCount:  0,
+    Timestamp: time.Now().UTC(),
+    Message:   "Seed: Task 'title' claimed. Description: ...",
+}
+
+hmmmRouter.Publish(ctx, seedMsg)
+```
+
+**Purpose:**
+- Creates dedicated discussion space for task
+- Enables agents to coordinate on specific tasks
+- Integrates with broader meta-coordination system
+- Provides context for SLURP event generation
+
+### Help Offer Reflection
+
+When agents offer help, the offer is reflected into the HMMM room:
+
+```go
+hmsg := hmmm.Message{
+    Version:   1,
+    Type:      "meta_msg",
+    IssueID:   issueID,
+    ThreadID:  fmt.Sprintf("issue-%d", issueID),
+    MsgID:     uuid.New().String(),
+    NodeID:    nodeID,
+    HopCount:  0,
+    Timestamp: time.Now().UTC(),
+    Message:   fmt.Sprintf("Help offer from %s (availability %d)",
+                          agentRole, availableSlots),
+}
+```
+
+## Availability Tracking
+
+The coordinator tracks task progress to keep availability broadcasts accurate:
+
+```go
+// When task is claimed:
+if tc.taskTracker != nil {
+    tc.taskTracker.AddTask(taskKey)
+}
+
+// When task completes:
+if tc.taskTracker != nil {
+    tc.taskTracker.RemoveTask(taskKey)
+}
+```
+
+This ensures the availability broadcaster (in `internal/runtime`) has accurate real-time data:
+
+```json
+{
+  "type": "availability_broadcast",
+  "data": {
+    "node_id": "Qm...",
+    "available_for_work": true,
+    "current_tasks": 1,
+    "max_tasks": 3,
+    "last_activity": 1727692800,
+    "status": "working",
+    "timestamp": 1727692800
+  }
+}
+```
+
+## Task Assignment Algorithm
+
+### Scoring System
+
+The `TaskMatcher` scores tasks for agents based on multiple factors:
+
+```
+Score = (roleMatch * 0.4) +
+        (expertiseMatch * 0.3) +
+        (availabilityScore * 0.2) +
+        (performanceScore * 0.1)
+
+Where:
+- roleMatch: 1.0 if agent role matches required role, 0.5 for partial match
+- expertiseMatch: percentage of required expertise agent possesses
+- availabilityScore: (maxTasks - currentTasks) / maxTasks
+- performanceScore: agent's historical performance metric (0.0-1.0)
+```
+
+**Threshold**: Tasks with score > 0.5 are considered for assignment.
+
+### Assignment Priority
+
+Tasks are prioritized by:
+1. **Priority Level** (task.Priority field, 0-10)
+2. **Task Score** (calculated by matcher)
+3. **Age** (older tasks first)
+4. **Dependencies** (tasks blocking others)
+
+### Claim Race Condition Handling
+
+Multiple agents may attempt to claim the same task:
+
+```
+1. Agent A evaluates task: score = 0.8, attempts claim
+2. Agent B evaluates task: score = 0.7, attempts claim
+3. Repository provider uses atomic claim operation
+4. First successful claim wins
+5. Other agents receive claim failure
+6. Failed agents continue to next task
+```
+
+## Error Handling
+
+### Task Execution Failures
+
+```go
+// On AI execution failure:
+if err := tc.executeTaskWithAI(activeTask); err != nil {
+    // Fall back to mock execution
+    taskResult = tc.executeMockTask(activeTask)
+}
+
+// On completion failure:
+if err := provider.CompleteTask(task, result); err != nil {
+    // Update status to failed
+    activeTask.Status = "failed"
+    activeTask.Results = map[string]interface{}{
+        "error": err.Error(),
+    }
+}
+```
+
+### Collaboration Request Failures
+
+```go
+err := tc.pubsub.PublishRoleBasedMessage(
+    pubsub.TaskHelpRequest, data, opts)
+if err != nil {
+    // Log error but continue with task
+    fmt.Printf("⚠️ Failed to request collaboration: %v\n", err)
+    // Task execution proceeds without collaboration
+}
+```
+
+### HMMM Seeding Failures
+
+```go
+if err := tc.hmmmRouter.Publish(ctx, seedMsg); err != nil {
+    // Log error to Hypercore
+    tc.hlog.AppendString("system_error", map[string]interface{}{
+        "error":       "hmmm_seed_failed",
+        "task_number": taskNumber,
+        "repository":  repository,
+        "message":     err.Error(),
+    })
+    // Task execution continues without HMMM room
+}
+```
+
+## Agent Configuration
+
+### Required Configuration
+
+```yaml
+agent:
+  id: "agent-001"
+  role: "Senior Backend Developer"
+  expertise:
+    - "Go"
+    - "PostgreSQL"
+    - "Docker"
+    - "Kubernetes"
+  capabilities:
+    - "code"
+    - "test"
+    - "deploy"
+  max_tasks: 3
+  specialization: "microservices"
+  models:
+    - name: "llama3.1:70b"
+      provider: "ollama"
+      endpoint: "http://192.168.1.72:11434"
+```
+
+### AgentInfo Structure
+
+```go
+type AgentInfo struct {
+    ID           string
+    Role         string
+    Expertise    []string
+    CurrentTasks int
+    MaxTasks     int
+    Status       string // ready, working, busy, offline
+    LastSeen     time.Time
+    Performance  map[string]interface{} // score: 0.8
+    Availability string // available, busy, offline
+}
+```
+
+## Hypercore Logging
+
+All coordination events are logged to Hypercore:
+
+### Task Claimed
+```go
+hlog.Append(logging.TaskClaimed, map[string]interface{}{
+    "task_number":   taskNumber,
+    "repository":    repository,
+    "title":         title,
+    "required_role": requiredRole,
+    "priority":      priority,
+})
+```
+
+### Task Completed
+```go
+hlog.Append(logging.TaskCompleted, map[string]interface{}{
+    "task_number": taskNumber,
+    "repository":  repository,
+    "duration":    durationSeconds,
+    "results":     resultsMap,
+})
+```
+
+## Status Reporting
+
+### Coordinator Status
+
+```go
+status := coordinator.GetStatus()
+// Returns:
+{
+    "agent_id":         "agent-001",
+    "role":             "Senior Backend Developer",
+    "expertise":        ["Go", "PostgreSQL", "Docker"],
+    "current_tasks":    1,
+    "max_tasks":        3,
+    "active_providers": 2,
+    "status":           "working",
+    "active_tasks": [
+        {
+            "repository": "myrepo",
+            "number":     42,
+            "title":      "Add authentication",
+            "status":     "working",
+            "claimed_at": "2025-09-30T10:00:00Z"
+        }
+    ]
+}
+```
+
+## Best Practices
+
+### Task Coordinator Usage
+
+1. **Initialize Early**: Create coordinator during agent startup
+2. **Set Task Tracker**: Always provide TaskProgressTracker for accurate availability
+3. **Configure HMMM**: Wire up hmmmRouter for meta-discussion integration
+4. **Monitor Status**: Periodically check GetStatus() for health monitoring
+5. **Handle Failures**: Implement proper error handling for degraded operation
+
+### Configuration Tuning
+
+1. **Max Tasks**: Set based on agent resources (CPU, memory, AI model capacity)
+2. **Sync Interval**: Balance between responsiveness and network overhead (default: 30s)
+3. **Task Scoring**: Adjust threshold (default: 0.5) based on task availability
+4. **Collaboration**: Enable for high-priority or expertise-gap tasks
+
+### Performance Optimization
+
+1. **Task Discovery**: Delegate to WHOOSH for efficient search and indexing
+2. **Concurrent Execution**: Use goroutines for parallel task execution
+3. **Lock Granularity**: Minimize lock contention with separate locks for providers/tasks
+4. **Caching**: Cache agent info and provider connections
+
+## Integration Points
+
+### With PubSub
+- Publishes: RoleAnnouncement, TaskProgress, TaskHelpRequest
+- Subscribes: TaskHelpRequest, ExpertiseRequest, CoordinationRequest
+- Topics: CHORUS/coordination/v1, hmmm/meta-discussion/v1
+
+### With HMMM
+- Seeds per-issue discussion rooms
+- Reflects help offers into rooms
+- Enables agent coordination on specific tasks
+
+### With Repository Providers
+- Claims tasks atomically
+- Fetches task details
+- Updates task status
+- Completes tasks with results
+
+### With Execution Engine
+- Converts repository tasks to execution requests
+- Executes tasks with AI providers
+- Handles sandbox environments
+- Collects execution metrics and artifacts
+
+### With Hypercore
+- Logs task claims
+- Logs task completions
+- Logs coordination errors
+- Provides audit trail
+
+## Task Message Format
+
+### PubSub Task Messages
+
+All task-related messages follow the standard PubSub Message format:
+
+```go
+type Message struct {
+    Type              MessageType            // e.g., "task_progress"
+    From              string                 // Peer ID
+    Timestamp         time.Time
+    Data              map[string]interface{} // Message payload
+    HopCount          int
+    FromRole          string                 // Agent role
+    ToRoles           []string               // Target roles
+    RequiredExpertise []string               // Required expertise
+    ProjectID         string
+    Priority          string                 // low, medium, high, urgent
+    ThreadID          string                 // Conversation thread
+}
+```
+
+### Task Assignment Message Flow
+
+```
+1. TaskAnnouncement (WHOOSH → PubSub)
+   ├─ Available task discovered
+   └─ Broadcast to coordination topic
+
+2. Task Evaluation (Local)
+   ├─ Score task for agent
+   └─ Decide whether to claim
+
+3. TaskClaim (Agent → Repository)
+   ├─ Atomic claim operation
+   └─ Only one agent succeeds
+
+4. TaskProgress (Agent → PubSub)
+   ├─ Announce claim to network
+   └─ Status: "claimed"
+
+5. TaskHelpRequest (Optional, Agent → PubSub)
+   ├─ Request collaboration if needed
+   └─ Target specific roles/expertise
+
+6. TaskHelpResponse (Other Agents → PubSub)
+   ├─ Offer assistance
+   └─ Include availability info
+
+7. TaskProgress (Agent → PubSub)
+   ├─ Announce work started
+   └─ Status: "started"
+
+8. Task Execution (Local with AI Engine)
+   ├─ Execute task in sandbox
+   └─ Generate artifacts
+
+9. TaskProgress (Agent → PubSub)
+   ├─ Announce completion
+   └─ Status: "completed"
+```
+
+## See Also
+
+- [discovery/](discovery.md) - mDNS peer discovery for local network
+- [pkg/coordination/](coordination.md) - Coordination primitives and dependency detection
+- [pubsub/](../pubsub.md) - PubSub messaging system
+- [pkg/execution/](execution.md) - Task execution engine
+- [pkg/hmmm/](hmmm.md) - Meta-discussion and coordination
+- [internal/runtime](../internal/runtime.md) - Agent runtime and availability broadcasting

docs/comprehensive/packages/discovery.md

@@ -0,0 +1,596 @@
+# Package: discovery
+
+**Location**: `/home/tony/chorus/project-queues/active/CHORUS/discovery/`
+
+## Overview
+
+The `discovery` package provides **mDNS-based peer discovery** for automatic detection and connection of CHORUS agents on the local network. It enables zero-configuration peer discovery using multicast DNS (mDNS), allowing agents to find and connect to each other without manual configuration or central coordination.
+
+## Architecture
+
+### mDNS Overview
+
+Multicast DNS (mDNS) is a protocol that resolves hostnames to IP addresses within small networks that do not include a local name server. It uses:
+
+- **Multicast IP**: 224.0.0.251 (IPv4) or FF02::FB (IPv6)
+- **UDP Port**: 5353
+- **Service Discovery**: Advertises and discovers services on the local network
+
+### CHORUS Service Tag
+
+**Default Service Name**: `"CHORUS-peer-discovery"`
+
+This service tag identifies CHORUS peers on the network. All CHORUS agents advertise themselves with this tag and listen for other agents using the same tag.
+
+## Core Components
+
+### MDNSDiscovery
+
+Main structure managing mDNS discovery operations.
+
+```go
+type MDNSDiscovery struct {
+    host        host.Host                 // libp2p host
+    service     mdns.Service              // mDNS service
+    notifee     *mdnsNotifee             // Peer notification handler
+    ctx         context.Context           // Discovery context
+    cancel      context.CancelFunc        // Context cancellation
+    serviceTag  string                    // Service name (default: "CHORUS-peer-discovery")
+}
+```
+
+**Key Responsibilities:**
+- Advertise local agent as mDNS service
+- Listen for mDNS announcements from other agents
+- Automatically connect to discovered peers
+- Handle peer connection lifecycle
+
+### mdnsNotifee
+
+Internal notification handler for discovered peers.
+
+```go
+type mdnsNotifee struct {
+    h         host.Host                // libp2p host
+    ctx       context.Context          // Context for operations
+    peersChan chan peer.AddrInfo       // Channel for discovered peers (buffer: 10)
+}
+```
+
+Implements the mDNS notification interface to receive peer discovery events.
+
+## Discovery Flow
+
+### 1. Service Initialization
+
+```go
+discovery, err := NewMDNSDiscovery(ctx, host, "CHORUS-peer-discovery")
+if err != nil {
+    return fmt.Errorf("failed to start mDNS discovery: %w", err)
+}
+```
+
+**Initialization Steps:**
+1. Create discovery context with cancellation
+2. Initialize mdnsNotifee with peer channel
+3. Create mDNS service with service tag
+4. Start mDNS service (begins advertising and listening)
+5. Launch background peer connection handler
+
+### 2. Service Advertisement
+
+When the service starts, it automatically advertises:
+
+```
+Service Type: _CHORUS-peer-discovery._udp.local
+Port: libp2p host port
+Addresses: All local IP addresses (IPv4 and IPv6)
+```
+
+This allows other CHORUS agents on the network to discover this peer.
+
+### 3. Peer Discovery
+
+**Discovery Process:**
+
+```
+1. mDNS Service listens for multicast announcements
+   ├─ Receives service announcement from peer
+   └─ Extracts peer.AddrInfo (ID + addresses)
+
+2. mdnsNotifee.HandlePeerFound() called
+   ├─ Peer info sent to peersChan
+   └─ Non-blocking send (drops if channel full)
+
+3. handleDiscoveredPeers() goroutine receives
+   ├─ Skip if peer is self
+   ├─ Skip if already connected
+   └─ Attempt connection
+```
+
+### 4. Automatic Connection
+
+```go
+func (d *MDNSDiscovery) handleDiscoveredPeers() {
+    for {
+        select {
+        case <-d.ctx.Done():
+            return
+        case peerInfo := <-d.notifee.peersChan:
+            // Skip self
+            if peerInfo.ID == d.host.ID() {
+                continue
+            }
+
+            // Check if already connected
+            if d.host.Network().Connectedness(peerInfo.ID) == 1 {
+                continue
+            }
+
+            // Attempt connection with timeout
+            connectCtx, cancel := context.WithTimeout(d.ctx, 10*time.Second)
+            err := d.host.Connect(connectCtx, peerInfo)
+            cancel()
+
+            if err != nil {
+                fmt.Printf("❌ Failed to connect to peer %s: %v\n",
+                          peerInfo.ID.ShortString(), err)
+            } else {
+                fmt.Printf("✅ Successfully connected to peer %s\n",
+                          peerInfo.ID.ShortString())
+            }
+        }
+    }
+}
+```
+
+**Connection Features:**
+- **10-second timeout** per connection attempt
+- **Idempotent**: Safe to attempt connection to already-connected peer
+- **Self-filtering**: Ignores own mDNS announcements
+- **Duplicate filtering**: Checks existing connections before attempting
+- **Non-blocking**: Runs in background goroutine
+
+## Usage
+
+### Basic Usage
+
+```go
+import (
+    "context"
+    "chorus/discovery"
+    "github.com/libp2p/go-libp2p/core/host"
+)
+
+func setupDiscovery(ctx context.Context, h host.Host) (*discovery.MDNSDiscovery, error) {
+    // Start mDNS discovery with default service tag
+    disc, err := discovery.NewMDNSDiscovery(ctx, h, "")
+    if err != nil {
+        return nil, err
+    }
+
+    fmt.Println("🔍 mDNS discovery started")
+    return disc, nil
+}
+```
+
+### Custom Service Tag
+
+```go
+// Use custom service tag for specific environments
+disc, err := discovery.NewMDNSDiscovery(ctx, h, "CHORUS-dev-network")
+if err != nil {
+    return nil, err
+}
+```
+
+### Monitoring Discovered Peers
+
+```go
+// Access peer channel for custom handling
+peersChan := disc.PeersChan()
+
+go func() {
+    for peerInfo := range peersChan {
+        fmt.Printf("🔍 Discovered peer: %s with %d addresses\n",
+                  peerInfo.ID.ShortString(),
+                  len(peerInfo.Addrs))
+
+        // Custom peer processing
+        handleNewPeer(peerInfo)
+    }
+}()
+```
+
+### Graceful Shutdown
+
+```go
+// Close discovery service
+if err := disc.Close(); err != nil {
+    log.Printf("Error closing discovery: %v", err)
+}
+```
+
+## Peer Information Structure
+
+### peer.AddrInfo
+
+Discovered peers are represented as libp2p `peer.AddrInfo`:
+
+```go
+type AddrInfo struct {
+    ID    peer.ID           // Unique peer identifier
+    Addrs []multiaddr.Multiaddr  // Peer addresses
+}
+```
+
+**Example Multiaddresses:**
+```
+/ip4/192.168.1.100/tcp/4001/p2p/QmPeerID...
+/ip6/fe80::1/tcp/4001/p2p/QmPeerID...
+```
+
+## Network Configuration
+
+### Firewall Requirements
+
+mDNS requires the following ports to be open:
+
+- **UDP 5353**: mDNS multicast
+- **TCP/UDP 4001** (or configured libp2p port): libp2p connections
+
+### Network Scope
+
+mDNS operates on **local network** only:
+- Same subnet required for discovery
+- Does not traverse routers (by design)
+- Ideal for LAN-based agent clusters
+
+### Multicast Group
+
+mDNS uses standard multicast groups:
+- **IPv4**: 224.0.0.251
+- **IPv6**: FF02::FB
+
+## Integration with CHORUS
+
+### Cluster Formation
+
+mDNS discovery enables automatic cluster formation:
+
+```
+Startup Sequence:
+1. Agent starts with libp2p host
+2. mDNS discovery initialized
+3. Agent advertises itself via mDNS
+4. Agent listens for other agents
+5. Auto-connects to discovered peers
+6. PubSub gossip network forms
+7. Task coordination begins
+```
+
+### Multi-Node Cluster Example
+
+```
+Network: 192.168.1.0/24
+
+Node 1 (walnut):     192.168.1.27  - Agent: backend-dev
+Node 2 (ironwood):   192.168.1.72  - Agent: frontend-dev
+Node 3 (rosewood):   192.168.1.113 - Agent: devops-specialist
+
+Discovery Flow:
+1. All nodes start with CHORUS-peer-discovery tag
+2. Each node multicasts to 224.0.0.251:5353
+3. All nodes receive each other's announcements
+4. Automatic connection establishment:
+   walnut ↔ ironwood
+   walnut ↔ rosewood
+   ironwood ↔ rosewood
+5. Full mesh topology formed
+6. PubSub topics synchronized
+```
+
+## Error Handling
+
+### Service Start Failure
+
+```go
+disc, err := discovery.NewMDNSDiscovery(ctx, h, serviceTag)
+if err != nil {
+    // Common causes:
+    // - Port 5353 already in use
+    // - Insufficient permissions (require multicast)
+    // - Network interface unavailable
+    return fmt.Errorf("failed to start mDNS discovery: %w", err)
+}
+```
+
+### Connection Failures
+
+Connection failures are logged but do not stop the discovery process:
+
+```
+❌ Failed to connect to peer Qm... : context deadline exceeded
+```
+
+**Common Causes:**
+- Peer behind firewall
+- Network congestion
+- Peer offline/restarting
+- Connection limit reached
+
+**Behavior**: Discovery continues, will retry on next mDNS announcement.
+
+### Channel Full
+
+If peer discovery is faster than connection handling:
+
+```
+⚠️ Discovery channel full, skipping peer Qm...
+```
+
+**Buffer Size**: 10 peers
+**Mitigation**: Non-critical, peer will be rediscovered on next announcement cycle
+
+## Performance Characteristics
+
+### Discovery Latency
+
+- **Initial Advertisement**: ~1-2 seconds after service start
+- **Discovery Response**: Typically < 1 second on LAN
+- **Connection Establishment**: 1-10 seconds (with 10s timeout)
+- **Re-announcement**: Periodic (standard mDNS timing)
+
+### Resource Usage
+
+- **Memory**: Minimal (~1MB per discovery service)
+- **CPU**: Very low (event-driven)
+- **Network**: Minimal (periodic multicast announcements)
+- **Concurrent Connections**: Handled by libp2p connection manager
+
+## Configuration Options
+
+### Service Tag Customization
+
+```go
+// Production environment
+disc, _ := discovery.NewMDNSDiscovery(ctx, h, "CHORUS-production")
+
+// Development environment
+disc, _ := discovery.NewMDNSDiscovery(ctx, h, "CHORUS-dev")
+
+// Testing environment
+disc, _ := discovery.NewMDNSDiscovery(ctx, h, "CHORUS-test")
+```
+
+**Use Case**: Isolate environments on same physical network.
+
+### Connection Timeout Adjustment
+
+Currently hardcoded to 10 seconds. For customization:
+
+```go
+// In handleDiscoveredPeers():
+connectTimeout := 30 * time.Second  // Longer for slow networks
+connectCtx, cancel := context.WithTimeout(d.ctx, connectTimeout)
+```
+
+## Advanced Usage
+
+### Custom Peer Handling
+
+Bypass automatic connection and implement custom logic:
+
+```go
+// Subscribe to peer channel
+peersChan := disc.PeersChan()
+
+go func() {
+    for peerInfo := range peersChan {
+        // Custom filtering
+        if shouldConnectToPeer(peerInfo) {
+            // Custom connection logic
+            connectWithRetry(peerInfo)
+        }
+    }
+}()
+```
+
+### Discovery Metrics
+
+```go
+type DiscoveryMetrics struct {
+    PeersDiscovered   int
+    ConnectionsSuccess int
+    ConnectionsFailed  int
+    LastDiscovery     time.Time
+}
+
+// Track metrics
+var metrics DiscoveryMetrics
+
+// In handleDiscoveredPeers():
+metrics.PeersDiscovered++
+if err := host.Connect(ctx, peerInfo); err != nil {
+    metrics.ConnectionsFailed++
+} else {
+    metrics.ConnectionsSuccess++
+}
+metrics.LastDiscovery = time.Now()
+```
+
+## Comparison with Other Discovery Methods
+
+### mDNS vs DHT
+
+| Feature | mDNS | DHT (Kademlia) |
+|---------|------|----------------|
+| Network Scope | Local network only | Global |
+| Setup | Zero-config | Requires bootstrap nodes |
+| Speed | Very fast (< 1s) | Slower (seconds to minutes) |
+| Privacy | Local only | Public network |
+| Reliability | High on LAN | Depends on DHT health |
+| Use Case | LAN clusters | Internet-wide P2P |
+
+**CHORUS Choice**: mDNS for local agent clusters, DHT could be added for internet-wide coordination.
+
+### mDNS vs Bootstrap List
+
+| Feature | mDNS | Bootstrap List |
+|---------|------|----------------|
+| Configuration | None | Manual list |
+| Maintenance | Automatic | Manual updates |
+| Scalability | Limited to LAN | Unlimited |
+| Flexibility | Dynamic | Static |
+| Failure Handling | Auto-discovery | Manual intervention |
+
+**CHORUS Choice**: mDNS for local discovery, bootstrap list as fallback.
+
+## libp2p Integration
+
+### Host Requirement
+
+mDNS discovery requires a libp2p host:
+
+```go
+import (
+    "github.com/libp2p/go-libp2p"
+    "github.com/libp2p/go-libp2p/core/host"
+)
+
+// Create libp2p host
+h, err := libp2p.New(
+    libp2p.ListenAddrStrings(
+        "/ip4/0.0.0.0/tcp/4001",
+        "/ip6/::/tcp/4001",
+    ),
+)
+if err != nil {
+    return err
+}
+
+// Initialize mDNS discovery with host
+disc, err := discovery.NewMDNSDiscovery(ctx, h, "CHORUS-peer-discovery")
+```
+
+### Connection Manager Integration
+
+mDNS discovery works with libp2p connection manager:
+
+```go
+h, err := libp2p.New(
+    libp2p.ListenAddrStrings("/ip4/0.0.0.0/tcp/4001"),
+    libp2p.ConnectionManager(connmgr.NewConnManager(
+        100,  // Low water mark
+        400,  // High water mark
+        time.Minute,
+    )),
+)
+
+// mDNS-discovered connections managed by connection manager
+disc, err := discovery.NewMDNSDiscovery(ctx, h, "")
+```
+
+## Security Considerations
+
+### Trust Model
+
+mDNS operates on **local network trust**:
+- Assumes local network is trusted
+- No authentication at mDNS layer
+- Authentication handled by libp2p security transport
+
+### Attack Vectors
+
+1. **Peer ID Spoofing**: Mitigated by libp2p peer ID verification
+2. **DoS via Fake Peers**: Limited by channel buffer and connection timeout
+3. **Network Snooping**: mDNS announcements are plaintext (by design)
+
+### Best Practices
+
+1. **Use libp2p Security**: TLS or Noise transport for encrypted connections
+2. **Peer Authentication**: Verify peer identities after connection
+3. **Network Isolation**: Deploy on trusted networks
+4. **Connection Limits**: Use libp2p connection manager
+5. **Monitoring**: Log all discovery and connection events
+
+## Troubleshooting
+
+### No Peers Discovered
+
+**Symptoms**: Service starts but no peers found.
+
+**Checks:**
+1. Verify all agents on same subnet
+2. Check firewall rules (UDP 5353)
+3. Verify mDNS/multicast not blocked by network
+4. Check service tag matches across agents
+5. Verify no mDNS conflicts with other services
+
+### Connection Failures
+
+**Symptoms**: Peers discovered but connections fail.
+
+**Checks:**
+1. Verify libp2p port open (default: TCP 4001)
+2. Check connection manager limits
+3. Verify peer addresses are reachable
+4. Check for NAT/firewall between peers
+5. Verify sufficient system resources (file descriptors, memory)
+
+### High CPU/Network Usage
+
+**Symptoms**: Excessive mDNS traffic or CPU usage.
+
+**Causes:**
+- Rapid peer restarts (re-announcements)
+- Many peers on network
+- Short announcement intervals
+
+**Solutions:**
+- Implement connection caching
+- Adjust mDNS announcement timing
+- Use connection limits
+
+## Monitoring and Debugging
+
+### Discovery Events
+
+```go
+// Log all discovery events
+disc, _ := discovery.NewMDNSDiscovery(ctx, h, "CHORUS-peer-discovery")
+
+peersChan := disc.PeersChan()
+go func() {
+    for peerInfo := range peersChan {
+        logger.Info("Discovered peer",
+            "peer_id", peerInfo.ID.String(),
+            "addresses", peerInfo.Addrs,
+            "timestamp", time.Now())
+    }
+}()
+```
+
+### Connection Status
+
+```go
+// Monitor connection status
+func monitorConnections(h host.Host) {
+    ticker := time.NewTicker(30 * time.Second)
+    defer ticker.Stop()
+
+    for range ticker.C {
+        peers := h.Network().Peers()
+        fmt.Printf("📊 Connected to %d peers: %v\n",
+                  len(peers), peers)
+    }
+}
+```
+
+## See Also
+
+- [coordinator/](coordinator.md) - Task coordination using discovered peers
+- [pubsub/](../pubsub.md) - PubSub over discovered peer network
+- [internal/runtime/](../internal/runtime.md) - Runtime initialization with discovery
+- [libp2p Documentation](https://docs.libp2p.io/) - libp2p concepts and APIs
+- [mDNS RFC 6762](https://tools.ietf.org/html/rfc6762) - mDNS protocol specification

docs/comprehensive/packages/metrics.md

@@ -0,0 +1,914 @@
+# CHORUS Metrics Package
+
+## Overview
+
+The `pkg/metrics` package provides comprehensive Prometheus-based metrics collection for the CHORUS distributed system. It exposes detailed operational metrics across all system components including P2P networking, DHT operations, PubSub messaging, elections, task management, and resource utilization.
+
+## Architecture
+
+### Core Components
+
+- **CHORUSMetrics**: Central metrics collector managing all Prometheus metrics
+- **Prometheus Registry**: Custom registry for metric collection
+- **HTTP Server**: Exposes metrics endpoint for scraping
+- **Background Collectors**: Periodic system and resource metric collection
+
+### Metric Types
+
+The package uses three Prometheus metric types:
+
+1. **Counter**: Monotonically increasing values (e.g., total messages sent)
+2. **Gauge**: Values that can go up or down (e.g., connected peers)
+3. **Histogram**: Distribution of values with configurable buckets (e.g., latency measurements)
+
+## Configuration
+
+### MetricsConfig
+
+```go
+type MetricsConfig struct {
+    // HTTP server configuration
+    ListenAddr  string        // Default: ":9090"
+    MetricsPath string        // Default: "/metrics"
+
+    // Histogram buckets
+    LatencyBuckets []float64  // Default: 0.001s to 10s
+    SizeBuckets    []float64  // Default: 64B to 16MB
+
+    // Node identification labels
+    NodeID      string        // Unique node identifier
+    Version     string        // CHORUS version
+    Environment string        // deployment environment (dev/staging/prod)
+    Cluster     string        // cluster identifier
+
+    // Collection intervals
+    SystemMetricsInterval   time.Duration  // Default: 30s
+    ResourceMetricsInterval time.Duration  // Default: 15s
+}
+```
+
+### Default Configuration
+
+```go
+config := metrics.DefaultMetricsConfig()
+// Returns:
+// - ListenAddr: ":9090"
+// - MetricsPath: "/metrics"
+// - LatencyBuckets: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
+// - SizeBuckets: [64, 256, 1024, 4096, 16384, 65536, 262144, 1048576, 4194304, 16777216]
+// - SystemMetricsInterval: 30s
+// - ResourceMetricsInterval: 15s
+```
+
+## Metrics Catalog
+
+### System Metrics
+
+#### chorus_system_info
+**Type**: Gauge
+**Description**: System information with version labels
+**Labels**: `node_id`, `version`, `go_version`, `cluster`, `environment`
+**Value**: Always 1 when present
+
+#### chorus_uptime_seconds
+**Type**: Gauge
+**Description**: System uptime in seconds since start
+**Value**: Current uptime in seconds
+
+### P2P Network Metrics
+
+#### chorus_p2p_connected_peers
+**Type**: Gauge
+**Description**: Number of currently connected P2P peers
+**Value**: Current peer count
+
+#### chorus_p2p_messages_sent_total
+**Type**: Counter
+**Description**: Total number of P2P messages sent
+**Labels**: `message_type`, `peer_id`
+**Usage**: Track outbound message volume per type and destination
+
+#### chorus_p2p_messages_received_total
+**Type**: Counter
+**Description**: Total number of P2P messages received
+**Labels**: `message_type`, `peer_id`
+**Usage**: Track inbound message volume per type and source
+
+#### chorus_p2p_message_latency_seconds
+**Type**: Histogram
+**Description**: P2P message round-trip latency distribution
+**Labels**: `message_type`
+**Buckets**: Configurable latency buckets (default: 1ms to 10s)
+
+#### chorus_p2p_connection_duration_seconds
+**Type**: Histogram
+**Description**: Duration of P2P connections
+**Labels**: `peer_id`
+**Usage**: Track connection stability
+
+#### chorus_p2p_peer_score
+**Type**: Gauge
+**Description**: Peer quality score
+**Labels**: `peer_id`
+**Value**: Score between 0.0 (poor) and 1.0 (excellent)
+
+### DHT (Distributed Hash Table) Metrics
+
+#### chorus_dht_put_operations_total
+**Type**: Counter
+**Description**: Total number of DHT put operations
+**Labels**: `status` (success/failure)
+**Usage**: Track DHT write operations
+
+#### chorus_dht_get_operations_total
+**Type**: Counter
+**Description**: Total number of DHT get operations
+**Labels**: `status` (success/failure)
+**Usage**: Track DHT read operations
+
+#### chorus_dht_operation_latency_seconds
+**Type**: Histogram
+**Description**: DHT operation latency distribution
+**Labels**: `operation` (put/get), `status` (success/failure)
+**Usage**: Monitor DHT performance
+
+#### chorus_dht_provider_records
+**Type**: Gauge
+**Description**: Number of provider records stored in DHT
+**Value**: Current provider record count
+
+#### chorus_dht_content_keys
+**Type**: Gauge
+**Description**: Number of content keys stored in DHT
+**Value**: Current content key count
+
+#### chorus_dht_replication_factor
+**Type**: Gauge
+**Description**: Replication factor for DHT keys
+**Labels**: `key_hash`
+**Value**: Number of replicas for specific keys
+
+#### chorus_dht_cache_hits_total
+**Type**: Counter
+**Description**: DHT cache hit count
+**Labels**: `cache_type`
+**Usage**: Monitor DHT caching effectiveness
+
+#### chorus_dht_cache_misses_total
+**Type**: Counter
+**Description**: DHT cache miss count
+**Labels**: `cache_type`
+**Usage**: Monitor DHT caching effectiveness
+
+### PubSub Messaging Metrics
+
+#### chorus_pubsub_topics
+**Type**: Gauge
+**Description**: Number of active PubSub topics
+**Value**: Current topic count
+
+#### chorus_pubsub_subscribers
+**Type**: Gauge
+**Description**: Number of subscribers per topic
+**Labels**: `topic`
+**Value**: Subscriber count for each topic
+
+#### chorus_pubsub_messages_total
+**Type**: Counter
+**Description**: Total PubSub messages
+**Labels**: `topic`, `direction` (sent/received), `message_type`
+**Usage**: Track message volume per topic
+
+#### chorus_pubsub_message_latency_seconds
+**Type**: Histogram
+**Description**: PubSub message delivery latency
+**Labels**: `topic`
+**Usage**: Monitor message propagation performance
+
+#### chorus_pubsub_message_size_bytes
+**Type**: Histogram
+**Description**: PubSub message size distribution
+**Labels**: `topic`
+**Buckets**: Configurable size buckets (default: 64B to 16MB)
+
+### Election System Metrics
+
+#### chorus_election_term
+**Type**: Gauge
+**Description**: Current election term number
+**Value**: Monotonically increasing term number
+
+#### chorus_election_state
+**Type**: Gauge
+**Description**: Current election state (1 for active state, 0 for others)
+**Labels**: `state` (idle/discovering/electing/reconstructing/complete)
+**Usage**: Only one state should have value 1 at any time
+
+#### chorus_heartbeats_sent_total
+**Type**: Counter
+**Description**: Total number of heartbeats sent by this node
+**Usage**: Monitor leader heartbeat activity
+
+#### chorus_heartbeats_received_total
+**Type**: Counter
+**Description**: Total number of heartbeats received from leader
+**Usage**: Monitor follower connectivity to leader
+
+#### chorus_leadership_changes_total
+**Type**: Counter
+**Description**: Total number of leadership changes
+**Usage**: Monitor election stability (lower is better)
+
+#### chorus_leader_uptime_seconds
+**Type**: Gauge
+**Description**: Current leader's tenure duration
+**Value**: Seconds since current leader was elected
+
+#### chorus_election_latency_seconds
+**Type**: Histogram
+**Description**: Time taken to complete election process
+**Usage**: Monitor election efficiency
+
+### Health Monitoring Metrics
+
+#### chorus_health_checks_passed_total
+**Type**: Counter
+**Description**: Total number of health checks passed
+**Labels**: `check_name`
+**Usage**: Track health check success rate
+
+#### chorus_health_checks_failed_total
+**Type**: Counter
+**Description**: Total number of health checks failed
+**Labels**: `check_name`, `reason`
+**Usage**: Track health check failures and reasons
+
+#### chorus_health_check_duration_seconds
+**Type**: Histogram
+**Description**: Health check execution duration
+**Labels**: `check_name`
+**Usage**: Monitor health check performance
+
+#### chorus_system_health_score
+**Type**: Gauge
+**Description**: Overall system health score
+**Value**: 0.0 (unhealthy) to 1.0 (healthy)
+**Usage**: Monitor overall system health
+
+#### chorus_component_health_score
+**Type**: Gauge
+**Description**: Component-specific health score
+**Labels**: `component`
+**Value**: 0.0 (unhealthy) to 1.0 (healthy)
+**Usage**: Track individual component health
+
+### Task Management Metrics
+
+#### chorus_tasks_active
+**Type**: Gauge
+**Description**: Number of currently active tasks
+**Value**: Current active task count
+
+#### chorus_tasks_queued
+**Type**: Gauge
+**Description**: Number of queued tasks waiting execution
+**Value**: Current queue depth
+
+#### chorus_tasks_completed_total
+**Type**: Counter
+**Description**: Total number of completed tasks
+**Labels**: `status` (success/failure), `task_type`
+**Usage**: Track task completion and success rate
+
+#### chorus_task_duration_seconds
+**Type**: Histogram
+**Description**: Task execution duration distribution
+**Labels**: `task_type`, `status`
+**Usage**: Monitor task performance
+
+#### chorus_task_queue_wait_time_seconds
+**Type**: Histogram
+**Description**: Time tasks spend in queue before execution
+**Usage**: Monitor task scheduling efficiency
+
+### SLURP (Context Generation) Metrics
+
+#### chorus_slurp_contexts_generated_total
+**Type**: Counter
+**Description**: Total number of SLURP contexts generated
+**Labels**: `role`, `status` (success/failure)
+**Usage**: Track context generation volume
+
+#### chorus_slurp_generation_time_seconds
+**Type**: Histogram
+**Description**: Time taken to generate SLURP contexts
+**Buckets**: [0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0, 120.0]
+**Usage**: Monitor context generation performance
+
+#### chorus_slurp_queue_length
+**Type**: Gauge
+**Description**: Length of SLURP generation queue
+**Value**: Current queue depth
+
+#### chorus_slurp_active_jobs
+**Type**: Gauge
+**Description**: Number of active SLURP generation jobs
+**Value**: Currently running generation jobs
+
+#### chorus_slurp_leadership_events_total
+**Type**: Counter
+**Description**: SLURP-related leadership events
+**Usage**: Track leader-initiated context generation
+
+### SHHH (Secret Sentinel) Metrics
+
+#### chorus_shhh_findings_total
+**Type**: Counter
+**Description**: Total number of SHHH redaction findings
+**Labels**: `rule`, `severity` (low/medium/high/critical)
+**Usage**: Monitor secret detection effectiveness
+
+### UCXI (Protocol Resolution) Metrics
+
+#### chorus_ucxi_requests_total
+**Type**: Counter
+**Description**: Total number of UCXI protocol requests
+**Labels**: `method`, `status` (success/failure)
+**Usage**: Track UCXI usage and success rate
+
+#### chorus_ucxi_resolution_latency_seconds
+**Type**: Histogram
+**Description**: UCXI address resolution latency
+**Usage**: Monitor resolution performance
+
+#### chorus_ucxi_cache_hits_total
+**Type**: Counter
+**Description**: UCXI cache hit count
+**Usage**: Monitor caching effectiveness
+
+#### chorus_ucxi_cache_misses_total
+**Type**: Counter
+**Description**: UCXI cache miss count
+**Usage**: Monitor caching effectiveness
+
+#### chorus_ucxi_content_size_bytes
+**Type**: Histogram
+**Description**: Size of resolved UCXI content
+**Usage**: Monitor content distribution
+
+### Resource Utilization Metrics
+
+#### chorus_cpu_usage_ratio
+**Type**: Gauge
+**Description**: CPU usage ratio
+**Value**: 0.0 (idle) to 1.0 (fully utilized)
+
+#### chorus_memory_usage_bytes
+**Type**: Gauge
+**Description**: Memory usage in bytes
+**Value**: Current memory consumption
+
+#### chorus_disk_usage_ratio
+**Type**: Gauge
+**Description**: Disk usage ratio
+**Labels**: `mount_point`
+**Value**: 0.0 (empty) to 1.0 (full)
+
+#### chorus_network_bytes_in_total
+**Type**: Counter
+**Description**: Total bytes received from network
+**Usage**: Track inbound network traffic
+
+#### chorus_network_bytes_out_total
+**Type**: Counter
+**Description**: Total bytes sent to network
+**Usage**: Track outbound network traffic
+
+#### chorus_goroutines
+**Type**: Gauge
+**Description**: Number of active goroutines
+**Value**: Current goroutine count
+
+### Error Metrics
+
+#### chorus_errors_total
+**Type**: Counter
+**Description**: Total number of errors
+**Labels**: `component`, `error_type`
+**Usage**: Track error frequency by component and type
+
+#### chorus_panics_total
+**Type**: Counter
+**Description**: Total number of panics recovered
+**Usage**: Monitor system stability
+
+## Usage Examples
+
+### Basic Initialization
+
+```go
+import "chorus/pkg/metrics"
+
+// Create metrics collector with default config
+config := metrics.DefaultMetricsConfig()
+config.NodeID = "chorus-node-01"
+config.Version = "v1.0.0"
+config.Environment = "production"
+config.Cluster = "cluster-01"
+
+metricsCollector := metrics.NewCHORUSMetrics(config)
+
+// Start metrics HTTP server
+if err := metricsCollector.StartServer(config); err != nil {
+    log.Fatalf("Failed to start metrics server: %v", err)
+}
+
+// Start background metric collection
+metricsCollector.CollectMetrics(config)
+```
+
+### Recording P2P Metrics
+
+```go
+// Update peer count
+metricsCollector.SetConnectedPeers(5)
+
+// Record message sent
+metricsCollector.IncrementMessagesSent("task_assignment", "peer-abc123")
+
+// Record message received
+metricsCollector.IncrementMessagesReceived("task_result", "peer-def456")
+
+// Record message latency
+startTime := time.Now()
+// ... send message and wait for response ...
+latency := time.Since(startTime)
+metricsCollector.ObserveMessageLatency("task_assignment", latency)
+```
+
+### Recording DHT Metrics
+
+```go
+// Record DHT put operation
+startTime := time.Now()
+err := dht.Put(key, value)
+latency := time.Since(startTime)
+
+if err != nil {
+    metricsCollector.IncrementDHTPutOperations("failure")
+    metricsCollector.ObserveDHTOperationLatency("put", "failure", latency)
+} else {
+    metricsCollector.IncrementDHTPutOperations("success")
+    metricsCollector.ObserveDHTOperationLatency("put", "success", latency)
+}
+
+// Update DHT statistics
+metricsCollector.SetDHTProviderRecords(150)
+metricsCollector.SetDHTContentKeys(450)
+metricsCollector.SetDHTReplicationFactor("key-hash-123", 3.0)
+```
+
+### Recording PubSub Metrics
+
+```go
+// Update topic count
+metricsCollector.SetPubSubTopics(10)
+
+// Record message published
+metricsCollector.IncrementPubSubMessages("CHORUS/tasks/v1", "sent", "task_created")
+
+// Record message received
+metricsCollector.IncrementPubSubMessages("CHORUS/tasks/v1", "received", "task_completed")
+
+// Record message latency
+startTime := time.Now()
+// ... publish message and wait for delivery confirmation ...
+latency := time.Since(startTime)
+metricsCollector.ObservePubSubMessageLatency("CHORUS/tasks/v1", latency)
+```
+
+### Recording Election Metrics
+
+```go
+// Update election state
+metricsCollector.SetElectionTerm(42)
+metricsCollector.SetElectionState("idle")
+
+// Record heartbeat sent (leader)
+metricsCollector.IncrementHeartbeatsSent()
+
+// Record heartbeat received (follower)
+metricsCollector.IncrementHeartbeatsReceived()
+
+// Record leadership change
+metricsCollector.IncrementLeadershipChanges()
+```
+
+### Recording Health Metrics
+
+```go
+// Record health check success
+metricsCollector.IncrementHealthCheckPassed("database-connectivity")
+
+// Record health check failure
+metricsCollector.IncrementHealthCheckFailed("p2p-connectivity", "no_peers")
+
+// Update health scores
+metricsCollector.SetSystemHealthScore(0.95)
+metricsCollector.SetComponentHealthScore("dht", 0.98)
+metricsCollector.SetComponentHealthScore("pubsub", 0.92)
+```
+
+### Recording Task Metrics
+
+```go
+// Update task counts
+metricsCollector.SetActiveTasks(5)
+metricsCollector.SetQueuedTasks(12)
+
+// Record task completion
+startTime := time.Now()
+// ... execute task ...
+duration := time.Since(startTime)
+
+metricsCollector.IncrementTasksCompleted("success", "data_processing")
+metricsCollector.ObserveTaskDuration("data_processing", "success", duration)
+```
+
+### Recording SLURP Metrics
+
+```go
+// Record context generation
+startTime := time.Now()
+// ... generate SLURP context ...
+duration := time.Since(startTime)
+
+metricsCollector.IncrementSLURPGenerated("admin", "success")
+metricsCollector.ObserveSLURPGenerationTime(duration)
+
+// Update queue length
+metricsCollector.SetSLURPQueueLength(3)
+```
+
+### Recording SHHH Metrics
+
+```go
+// Record secret findings
+findings := scanForSecrets(content)
+for _, finding := range findings {
+    metricsCollector.IncrementSHHHFindings(finding.Rule, finding.Severity, 1)
+}
+```
+
+### Recording Resource Metrics
+
+```go
+import "runtime"
+
+// Get runtime stats
+var memStats runtime.MemStats
+runtime.ReadMemStats(&memStats)
+
+metricsCollector.SetMemoryUsage(float64(memStats.Alloc))
+metricsCollector.SetGoroutines(runtime.NumGoroutine())
+
+// Record system resource usage
+metricsCollector.SetCPUUsage(0.45)  // 45% CPU usage
+metricsCollector.SetDiskUsage("/var/lib/CHORUS", 0.73)  // 73% disk usage
+```
+
+### Recording Errors
+
+```go
+// Record error occurrence
+if err != nil {
+    metricsCollector.IncrementErrors("dht", "timeout")
+}
+
+// Record recovered panic
+defer func() {
+    if r := recover(); r != nil {
+        metricsCollector.IncrementPanics()
+        // Handle panic...
+    }
+}()
+```
+
+## Prometheus Integration
+
+### Scrape Configuration
+
+Add the following to your `prometheus.yml`:
+
+```yaml
+scrape_configs:
+  - job_name: 'chorus-nodes'
+    scrape_interval: 15s
+    scrape_timeout: 10s
+    metrics_path: '/metrics'
+    static_configs:
+      - targets:
+          - 'chorus-node-01:9090'
+          - 'chorus-node-02:9090'
+          - 'chorus-node-03:9090'
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: instance
+      - source_labels: [__address__]
+        regex: '([^:]+):.*'
+        target_label: node
+        replacement: '${1}'
+```
+
+### Example Queries
+
+#### P2P Network Health
+```promql
+# Average connected peers across cluster
+avg(chorus_p2p_connected_peers)
+
+# Message rate per second
+rate(chorus_p2p_messages_sent_total[5m])
+
+# 95th percentile message latency
+histogram_quantile(0.95, rate(chorus_p2p_message_latency_seconds_bucket[5m]))
+```
+
+#### DHT Performance
+```promql
+# DHT operation success rate
+rate(chorus_dht_get_operations_total{status="success"}[5m]) /
+rate(chorus_dht_get_operations_total[5m])
+
+# Average DHT operation latency
+rate(chorus_dht_operation_latency_seconds_sum[5m]) /
+rate(chorus_dht_operation_latency_seconds_count[5m])
+
+# DHT cache hit rate
+rate(chorus_dht_cache_hits_total[5m]) /
+(rate(chorus_dht_cache_hits_total[5m]) + rate(chorus_dht_cache_misses_total[5m]))
+```
+
+#### Election Stability
+```promql
+# Leadership changes per hour
+rate(chorus_leadership_changes_total[1h]) * 3600
+
+# Nodes by election state
+sum by (state) (chorus_election_state)
+
+# Heartbeat rate
+rate(chorus_heartbeats_sent_total[5m])
+```
+
+#### Task Management
+```promql
+# Task success rate
+rate(chorus_tasks_completed_total{status="success"}[5m]) /
+rate(chorus_tasks_completed_total[5m])
+
+# Average task duration
+histogram_quantile(0.50, rate(chorus_task_duration_seconds_bucket[5m]))
+
+# Task queue depth
+chorus_tasks_queued
+```
+
+#### Resource Utilization
+```promql
+# CPU usage by node
+chorus_cpu_usage_ratio
+
+# Memory usage by node
+chorus_memory_usage_bytes / (1024 * 1024 * 1024)  # Convert to GB
+
+# Disk usage alert (>90%)
+chorus_disk_usage_ratio > 0.9
+```
+
+#### System Health
+```promql
+# Overall system health score
+chorus_system_health_score
+
+# Component health scores
+chorus_component_health_score
+
+# Health check failure rate
+rate(chorus_health_checks_failed_total[5m])
+```
+
+### Alerting Rules
+
+Example Prometheus alerting rules for CHORUS:
+
+```yaml
+groups:
+  - name: chorus_alerts
+    interval: 30s
+    rules:
+      # P2P connectivity alerts
+      - alert: LowPeerCount
+        expr: chorus_p2p_connected_peers < 2
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Low P2P peer count on {{ $labels.instance }}"
+          description: "Node has {{ $value }} peers (minimum: 2)"
+
+      # DHT performance alerts
+      - alert: HighDHTFailureRate
+        expr: |
+          rate(chorus_dht_get_operations_total{status="failure"}[5m]) /
+          rate(chorus_dht_get_operations_total[5m]) > 0.1
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High DHT failure rate on {{ $labels.instance }}"
+          description: "DHT failure rate: {{ $value | humanizePercentage }}"
+
+      # Election stability alerts
+      - alert: FrequentLeadershipChanges
+        expr: rate(chorus_leadership_changes_total[1h]) * 3600 > 5
+        for: 15m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Frequent leadership changes"
+          description: "{{ $value }} leadership changes per hour"
+
+      # Task management alerts
+      - alert: HighTaskQueueDepth
+        expr: chorus_tasks_queued > 100
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High task queue depth on {{ $labels.instance }}"
+          description: "{{ $value }} tasks queued"
+
+      # Resource alerts
+      - alert: HighMemoryUsage
+        expr: chorus_memory_usage_bytes > 8 * 1024 * 1024 * 1024  # 8GB
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High memory usage on {{ $labels.instance }}"
+          description: "Memory usage: {{ $value | humanize1024 }}B"
+
+      - alert: HighDiskUsage
+        expr: chorus_disk_usage_ratio > 0.9
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          summary: "High disk usage on {{ $labels.instance }}"
+          description: "Disk usage: {{ $value | humanizePercentage }}"
+
+      # Health monitoring alerts
+      - alert: LowSystemHealth
+        expr: chorus_system_health_score < 0.75
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Low system health score on {{ $labels.instance }}"
+          description: "Health score: {{ $value }}"
+
+      - alert: ComponentUnhealthy
+        expr: chorus_component_health_score < 0.5
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Component {{ $labels.component }} unhealthy"
+          description: "Health score: {{ $value }}"
+```
+
+## HTTP Endpoints
+
+### Metrics Endpoint
+
+**URL**: `/metrics`
+**Method**: GET
+**Description**: Prometheus metrics in text exposition format
+
+**Response Format**:
+```
+# HELP chorus_p2p_connected_peers Number of connected P2P peers
+# TYPE chorus_p2p_connected_peers gauge
+chorus_p2p_connected_peers 5
+
+# HELP chorus_dht_put_operations_total Total number of DHT put operations
+# TYPE chorus_dht_put_operations_total counter
+chorus_dht_put_operations_total{status="success"} 1523
+chorus_dht_put_operations_total{status="failure"} 12
+
+# HELP chorus_task_duration_seconds Task execution duration
+# TYPE chorus_task_duration_seconds histogram
+chorus_task_duration_seconds_bucket{task_type="data_processing",status="success",le="0.001"} 0
+chorus_task_duration_seconds_bucket{task_type="data_processing",status="success",le="0.005"} 12
+chorus_task_duration_seconds_bucket{task_type="data_processing",status="success",le="0.01"} 45
+...
+```
+
+### Health Endpoint
+
+**URL**: `/health`
+**Method**: GET
+**Description**: Basic health check for metrics server
+
+**Response**: `200 OK` with body `OK`
+
+## Best Practices
+
+### Metric Naming
+- Use descriptive metric names with `chorus_` prefix
+- Follow Prometheus naming conventions: `component_metric_unit`
+- Use `_total` suffix for counters
+- Use `_seconds` suffix for time measurements
+- Use `_bytes` suffix for size measurements
+
+### Label Usage
+- Keep label cardinality low (avoid high-cardinality labels like request IDs)
+- Use consistent label names across metrics
+- Document label meanings and expected values
+- Avoid labels that change frequently
+
+### Performance Considerations
+- Metrics collection is lock-free for read operations
+- Histogram observations are optimized for high throughput
+- Background collectors run on separate goroutines
+- Custom registry prevents pollution of default registry
+
+### Error Handling
+- Metrics collection should never panic
+- Failed metric updates should be logged but not block operations
+- Use nil checks before accessing metrics collectors
+
+### Testing
+```go
+func TestMetrics(t *testing.T) {
+    config := metrics.DefaultMetricsConfig()
+    config.NodeID = "test-node"
+
+    m := metrics.NewCHORUSMetrics(config)
+
+    // Test metric updates
+    m.SetConnectedPeers(5)
+    m.IncrementMessagesSent("test", "peer1")
+
+    // Verify metrics are collected
+    // (Use prometheus testutil for verification)
+}
+```
+
+## Troubleshooting
+
+### Metrics Not Appearing
+1. Verify metrics server is running: `curl http://localhost:9090/metrics`
+2. Check configuration: ensure correct `ListenAddr` and `MetricsPath`
+3. Verify Prometheus scrape configuration
+4. Check for errors in application logs
+
+### High Memory Usage
+1. Review label cardinality (check for unbounded label values)
+2. Adjust histogram buckets if too granular
+3. Reduce metric collection frequency
+4. Consider metric retention policies in Prometheus
+
+### Missing Metrics
+1. Ensure metric is being updated by application code
+2. Verify metric registration in `initializeMetrics()`
+3. Check for race conditions in metric access
+4. Review metric type compatibility (Counter vs Gauge vs Histogram)
+
+## Migration Guide
+
+### From Default Prometheus Registry
+```go
+// Old approach
+prometheus.MustRegister(myCounter)
+
+// New approach
+config := metrics.DefaultMetricsConfig()
+m := metrics.NewCHORUSMetrics(config)
+// Use m.IncrementErrors(...) instead of direct counter access
+```
+
+### Adding New Metrics
+1. Add metric field to `CHORUSMetrics` struct
+2. Initialize metric in `initializeMetrics()` method
+3. Add helper methods for updating the metric
+4. Document the metric in this file
+5. Add Prometheus queries and alerts as needed
+
+## Related Documentation
+
+- [Health Package Documentation](./health.md)
+- [Shutdown Package Documentation](./shutdown.md)
+- [Prometheus Documentation](https://prometheus.io/docs/)
+- [Prometheus Best Practices](https://prometheus.io/docs/practices/naming/)