docs: Add Phase 3 coordination and infrastructure documentation
Comprehensive documentation for coordination, messaging, discovery, and internal systems. Core Coordination Packages: - pkg/election - Democratic leader election (uptime-based, heartbeat mechanism, SLURP integration) - pkg/coordination - Meta-coordination with dependency detection (4 built-in rules) - coordinator/ - Task orchestration and assignment (AI-powered scoring) - discovery/ - mDNS peer discovery (automatic LAN detection) Messaging & P2P Infrastructure: - pubsub/ - GossipSub messaging (31 message types, role-based topics, HMMM integration) - p2p/ - libp2p networking (DHT modes, connection management, security) Monitoring & Health: - pkg/metrics - Prometheus metrics (80+ metrics across 12 categories) - pkg/health - Health monitoring (4 HTTP endpoints, enhanced checks, graceful degradation) Internal Systems: - internal/licensing - License validation (KACHING integration, cluster leases, fail-closed) - internal/hapui - Human Agent Portal UI (9 commands, HMMM wizard, UCXL browser, decision voting) - internal/backbeat - P2P operation telemetry (6 phases, beat synchronization, health reporting) Documentation Statistics (Phase 3): - 10 packages documented (~18,000 lines) - 31 PubSub message types cataloged - 80+ Prometheus metrics documented - Complete API references with examples - Integration patterns and best practices Key Features Documented: - Election: 5 triggers, candidate scoring (5 weighted components), stability windows - Coordination: AI-powered dependency detection, cross-repo sessions, escalation handling - PubSub: Topic patterns, message envelopes, SHHH redaction, Hypercore logging - Metrics: All metric types with labels, Prometheus scrape config, alert rules - Health: Liveness vs readiness, critical checks, Kubernetes integration - Licensing: Grace periods, circuit breaker, cluster lease management - HAP UI: Interactive terminal commands, HMMM composition wizard, web interface (beta) - BACKBEAT: 6-phase operation tracking, beat budget estimation, drift detection Implementation Status Marked: - ✅ Production: Election, metrics, health, licensing, pubsub, p2p, discovery, coordinator - 🔶 Beta: HAP web interface, BACKBEAT telemetry, advanced coordination - 🔷 Alpha: SLURP election scoring - ⚠️ Experimental: Meta-coordination, AI-powered dependency detection Progress: 22/62 files complete (35%) Next Phase: AI providers, SLURP system, API layer, reasoning engine 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
anthonyrawlins
949
docs/comprehensive/packages/coordination.md
Normal file
949
docs/comprehensive/packages/coordination.md
Normal file
@@ -0,0 +1,949 @@
|
||||
# Package: pkg/coordination
|
||||
|
||||
**Location**: `/home/tony/chorus/project-queues/active/CHORUS/pkg/coordination/`
|
||||
|
||||
## Overview
|
||||
|
||||
The `pkg/coordination` package provides **advanced cross-repository coordination primitives** for managing complex task dependencies and multi-agent collaboration in CHORUS. It includes AI-powered dependency detection, meta-coordination sessions, and automated escalation handling to enable sophisticated distributed development workflows.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Coordination Layers
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ MetaCoordinator │
|
||||
│ - Session management │
|
||||
│ - AI-powered coordination planning │
|
||||
│ - Escalation handling │
|
||||
│ - SLURP integration │
|
||||
└─────────────────┬───────────────────────────────┘
|
||||
│
|
||||
┌─────────────────▼───────────────────────────────┐
|
||||
│ DependencyDetector │
|
||||
│ - Cross-repo dependency detection │
|
||||
│ - Rule-based pattern matching │
|
||||
│ - Relationship analysis │
|
||||
└─────────────────┬───────────────────────────────┘
|
||||
│
|
||||
┌─────────────────▼───────────────────────────────┐
|
||||
│ PubSub (HMMM Meta-Discussion) │
|
||||
│ - Coordination messages │
|
||||
│ - Session broadcasts │
|
||||
│ - Escalation notifications │
|
||||
└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Core Components
|
||||
|
||||
### MetaCoordinator
|
||||
|
||||
Manages advanced cross-repository coordination and multi-agent collaboration sessions.
|
||||
|
||||
```go
|
||||
type MetaCoordinator struct {
|
||||
pubsub *pubsub.PubSub
|
||||
ctx context.Context
|
||||
dependencyDetector *DependencyDetector
|
||||
slurpIntegrator *integration.SlurpEventIntegrator
|
||||
|
||||
// Active coordination sessions
|
||||
activeSessions map[string]*CoordinationSession
|
||||
sessionLock sync.RWMutex
|
||||
|
||||
// Configuration
|
||||
maxSessionDuration time.Duration // Default: 30 minutes
|
||||
maxParticipants int // Default: 5
|
||||
escalationThreshold int // Default: 10 messages
|
||||
}
|
||||
```
|
||||
|
||||
**Key Responsibilities:**
|
||||
- Create and manage coordination sessions
|
||||
- Generate AI-powered coordination plans
|
||||
- Monitor session progress and health
|
||||
- Escalate to humans when needed
|
||||
- Generate SLURP events from coordination outcomes
|
||||
- Integrate with HMMM for meta-discussion
|
||||
|
||||
### DependencyDetector
|
||||
|
||||
Analyzes tasks across repositories to detect relationships and dependencies.
|
||||
|
||||
```go
|
||||
type DependencyDetector struct {
|
||||
pubsub *pubsub.PubSub
|
||||
ctx context.Context
|
||||
knownTasks map[string]*TaskContext
|
||||
dependencyRules []DependencyRule
|
||||
coordinationHops int // Default: 3
|
||||
}
|
||||
```
|
||||
|
||||
**Key Responsibilities:**
|
||||
- Track tasks across multiple repositories
|
||||
- Apply pattern-based dependency detection rules
|
||||
- Identify task relationships (API contracts, schema changes, etc.)
|
||||
- Broadcast dependency alerts
|
||||
- Trigger coordination sessions
|
||||
|
||||
### CoordinationSession
|
||||
|
||||
Represents an active multi-agent coordination session.
|
||||
|
||||
```go
|
||||
type CoordinationSession struct {
|
||||
SessionID string
|
||||
Type string // dependency, conflict, planning
|
||||
Participants map[string]*Participant
|
||||
TasksInvolved []*TaskContext
|
||||
Messages []CoordinationMessage
|
||||
Status string // active, resolved, escalated
|
||||
CreatedAt time.Time
|
||||
LastActivity time.Time
|
||||
Resolution string
|
||||
EscalationReason string
|
||||
}
|
||||
```
|
||||
|
||||
**Session Types:**
|
||||
- **dependency**: Coordinating dependent tasks across repos
|
||||
- **conflict**: Resolving conflicts or competing changes
|
||||
- **planning**: Joint planning for complex multi-repo features
|
||||
|
||||
**Session States:**
|
||||
- **active**: Session in progress
|
||||
- **resolved**: Consensus reached, coordination complete
|
||||
- **escalated**: Requires human intervention
|
||||
|
||||
## Data Structures
|
||||
|
||||
### TaskContext
|
||||
|
||||
Represents a task with its repository and project context for dependency analysis.
|
||||
|
||||
```go
|
||||
type TaskContext struct {
|
||||
TaskID int
|
||||
ProjectID int
|
||||
Repository string
|
||||
Title string
|
||||
Description string
|
||||
Keywords []string
|
||||
AgentID string
|
||||
ClaimedAt time.Time
|
||||
}
|
||||
```
|
||||
|
||||
### Participant
|
||||
|
||||
Represents an agent participating in a coordination session.
|
||||
|
||||
```go
|
||||
type Participant struct {
|
||||
AgentID string
|
||||
PeerID string
|
||||
Repository string
|
||||
Capabilities []string
|
||||
LastSeen time.Time
|
||||
Active bool
|
||||
}
|
||||
```
|
||||
|
||||
### CoordinationMessage
|
||||
|
||||
A message within a coordination session.
|
||||
|
||||
```go
|
||||
type CoordinationMessage struct {
|
||||
MessageID string
|
||||
FromAgentID string
|
||||
FromPeerID string
|
||||
Content string
|
||||
MessageType string // proposal, question, agreement, concern
|
||||
Timestamp time.Time
|
||||
Metadata map[string]interface{}
|
||||
}
|
||||
```
|
||||
|
||||
**Message Types:**
|
||||
- **proposal**: Proposed solution or approach
|
||||
- **question**: Request for clarification
|
||||
- **agreement**: Agreement with proposal
|
||||
- **concern**: Concern or objection
|
||||
|
||||
### TaskDependency
|
||||
|
||||
Represents a detected relationship between tasks.
|
||||
|
||||
```go
|
||||
type TaskDependency struct {
|
||||
Task1 *TaskContext
|
||||
Task2 *TaskContext
|
||||
Relationship string // Rule name (e.g., "API_Contract")
|
||||
Confidence float64 // 0.0 - 1.0
|
||||
Reason string // Human-readable explanation
|
||||
DetectedAt time.Time
|
||||
}
|
||||
```
|
||||
|
||||
### DependencyRule
|
||||
|
||||
Defines how to detect task relationships.
|
||||
|
||||
```go
|
||||
type DependencyRule struct {
|
||||
Name string
|
||||
Description string
|
||||
Keywords []string
|
||||
Validator func(task1, task2 *TaskContext) (bool, string)
|
||||
}
|
||||
```
|
||||
|
||||
## Dependency Detection
|
||||
|
||||
### Built-in Detection Rules
|
||||
|
||||
#### 1. API Contract Rule
|
||||
|
||||
Detects dependencies between API definitions and implementations.
|
||||
|
||||
```go
|
||||
{
|
||||
Name: "API_Contract",
|
||||
Description: "Tasks involving API contracts and implementations",
|
||||
Keywords: []string{"api", "endpoint", "contract", "interface", "schema"},
|
||||
Validator: func(task1, task2 *TaskContext) (bool, string) {
|
||||
text1 := strings.ToLower(task1.Title + " " + task1.Description)
|
||||
text2 := strings.ToLower(task2.Title + " " + task2.Description)
|
||||
|
||||
if (strings.Contains(text1, "api") && strings.Contains(text2, "implement")) ||
|
||||
(strings.Contains(text2, "api") && strings.Contains(text1, "implement")) {
|
||||
return true, "API definition and implementation dependency"
|
||||
}
|
||||
return false, ""
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**Example Detection:**
|
||||
- Task 1: "Define user authentication API"
|
||||
- Task 2: "Implement authentication endpoint"
|
||||
- **Detected**: API_Contract dependency
|
||||
|
||||
#### 2. Database Schema Rule
|
||||
|
||||
Detects schema changes affecting multiple services.
|
||||
|
||||
```go
|
||||
{
|
||||
Name: "Database_Schema",
|
||||
Description: "Database schema changes affecting multiple services",
|
||||
Keywords: []string{"database", "schema", "migration", "table", "model"},
|
||||
Validator: func(task1, task2 *TaskContext) (bool, string) {
|
||||
// Checks for database-related keywords in both tasks
|
||||
// Returns true if both tasks involve database work
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**Example Detection:**
|
||||
- Task 1: "Add user preferences table"
|
||||
- Task 2: "Update user service for preferences"
|
||||
- **Detected**: Database_Schema dependency
|
||||
|
||||
#### 3. Configuration Dependency Rule
|
||||
|
||||
Detects configuration changes affecting multiple components.
|
||||
|
||||
```go
|
||||
{
|
||||
Name: "Configuration_Dependency",
|
||||
Description: "Configuration changes affecting multiple components",
|
||||
Keywords: []string{"config", "environment", "settings", "parameters"},
|
||||
}
|
||||
```
|
||||
|
||||
**Example Detection:**
|
||||
- Task 1: "Add feature flag for new UI"
|
||||
- Task 2: "Implement feature flag checks in backend"
|
||||
- **Detected**: Configuration_Dependency
|
||||
|
||||
#### 4. Security Compliance Rule
|
||||
|
||||
Detects security changes requiring coordinated implementation.
|
||||
|
||||
```go
|
||||
{
|
||||
Name: "Security_Compliance",
|
||||
Description: "Security changes requiring coordinated implementation",
|
||||
Keywords: []string{"security", "auth", "permission", "token", "encrypt"},
|
||||
}
|
||||
```
|
||||
|
||||
**Example Detection:**
|
||||
- Task 1: "Implement JWT token refresh"
|
||||
- Task 2: "Update authentication middleware"
|
||||
- **Detected**: Security_Compliance dependency
|
||||
|
||||
### Custom Rules
|
||||
|
||||
Add project-specific dependency detection:
|
||||
|
||||
```go
|
||||
customRule := DependencyRule{
|
||||
Name: "GraphQL_Schema",
|
||||
Description: "GraphQL schema and resolver dependencies",
|
||||
Keywords: []string{"graphql", "schema", "resolver", "query", "mutation"},
|
||||
Validator: func(task1, task2 *TaskContext) (bool, string) {
|
||||
text1 := strings.ToLower(task1.Title + " " + task1.Description)
|
||||
text2 := strings.ToLower(task2.Title + " " + task2.Description)
|
||||
|
||||
hasSchema := strings.Contains(text1, "schema") || strings.Contains(text2, "schema")
|
||||
hasResolver := strings.Contains(text1, "resolver") || strings.Contains(text2, "resolver")
|
||||
|
||||
if hasSchema && hasResolver {
|
||||
return true, "GraphQL schema and resolver must be coordinated"
|
||||
}
|
||||
return false, ""
|
||||
},
|
||||
}
|
||||
|
||||
dependencyDetector.AddCustomRule(customRule)
|
||||
```
|
||||
|
||||
## Coordination Flow
|
||||
|
||||
### 1. Task Registration and Detection
|
||||
|
||||
```
|
||||
Task Claimed by Agent A → RegisterTask() → DependencyDetector
|
||||
↓
|
||||
detectDependencies()
|
||||
↓
|
||||
Apply all dependency rules to known tasks
|
||||
↓
|
||||
Dependency detected? → Yes → announceDependency()
|
||||
↓ ↓
|
||||
No MetaCoordinator
|
||||
```
|
||||
|
||||
### 2. Dependency Announcement
|
||||
|
||||
```go
|
||||
// Dependency detector announces to HMMM meta-discussion
|
||||
coordMsg := map[string]interface{}{
|
||||
"message_type": "dependency_detected",
|
||||
"dependency": dep,
|
||||
"coordination_request": "Cross-repository dependency detected...",
|
||||
"agents_involved": [agentA, agentB],
|
||||
"repositories": [repoA, repoB],
|
||||
"hop_count": 0,
|
||||
"max_hops": 3,
|
||||
}
|
||||
|
||||
pubsub.PublishHmmmMessage(MetaDiscussion, coordMsg)
|
||||
```
|
||||
|
||||
### 3. Session Creation
|
||||
|
||||
```
|
||||
MetaCoordinator receives dependency_detected message
|
||||
↓
|
||||
handleDependencyDetection()
|
||||
↓
|
||||
Create CoordinationSession
|
||||
↓
|
||||
Add participating agents
|
||||
↓
|
||||
Generate AI coordination plan
|
||||
↓
|
||||
Broadcast plan to participants
|
||||
```
|
||||
|
||||
### 4. AI-Powered Coordination Planning
|
||||
|
||||
```go
|
||||
prompt := `
|
||||
You are an expert AI project coordinator managing a distributed development team.
|
||||
|
||||
SITUATION:
|
||||
- A dependency has been detected between two tasks in different repositories
|
||||
- Task 1: repo1/title #42 (Agent: agent-001)
|
||||
- Task 2: repo2/title #43 (Agent: agent-002)
|
||||
- Relationship: API_Contract
|
||||
- Reason: API definition and implementation dependency
|
||||
|
||||
COORDINATION REQUIRED:
|
||||
Generate a concise coordination plan that addresses:
|
||||
1. What specific coordination is needed between the agents
|
||||
2. What order should tasks be completed in (if any)
|
||||
3. What information/artifacts need to be shared
|
||||
4. What potential conflicts to watch for
|
||||
5. Success criteria for coordinated completion
|
||||
`
|
||||
|
||||
plan := reasoning.GenerateResponse(ctx, "phi3", prompt)
|
||||
```
|
||||
|
||||
**Plan Output Example:**
|
||||
```
|
||||
COORDINATION PLAN:
|
||||
|
||||
1. SEQUENCE:
|
||||
- Task 1 (API definition) must be completed first
|
||||
- Task 2 (implementation) depends on finalized API contract
|
||||
|
||||
2. INFORMATION SHARING:
|
||||
- Agent-001 must share: API specification document, endpoint definitions
|
||||
- Agent-002 must share: Implementation plan, integration tests
|
||||
|
||||
3. COORDINATION POINTS:
|
||||
- Review API spec before implementation begins
|
||||
- Daily sync on implementation progress
|
||||
- Joint testing before completion
|
||||
|
||||
4. POTENTIAL CONFLICTS:
|
||||
- API spec changes during implementation
|
||||
- Performance requirements not captured in spec
|
||||
- Authentication/authorization approach
|
||||
|
||||
5. SUCCESS CRITERIA:
|
||||
- API spec reviewed and approved
|
||||
- Implementation matches spec
|
||||
- Integration tests pass
|
||||
- Documentation complete
|
||||
```
|
||||
|
||||
### 5. Session Progress Monitoring
|
||||
|
||||
```
|
||||
Agents respond to coordination plan
|
||||
↓
|
||||
handleCoordinationResponse()
|
||||
↓
|
||||
Add message to session
|
||||
↓
|
||||
Update participant activity
|
||||
↓
|
||||
evaluateSessionProgress()
|
||||
↓
|
||||
┌──────────────────────┐
|
||||
│ Check conditions: │
|
||||
│ - Message count │
|
||||
│ - Session duration │
|
||||
│ - Agreement keywords │
|
||||
└──────┬───────────────┘
|
||||
│
|
||||
┌──────▼──────┬──────────────┐
|
||||
│ │ │
|
||||
Consensus? Too long? Too many msgs?
|
||||
│ │ │
|
||||
Resolved Escalate Escalate
|
||||
```
|
||||
|
||||
### 6. Session Resolution
|
||||
|
||||
**Consensus Reached:**
|
||||
```go
|
||||
// Detect agreement in recent messages
|
||||
agreementKeywords := []string{
|
||||
"agree", "sounds good", "approved", "looks good", "confirmed"
|
||||
}
|
||||
|
||||
if agreementCount >= len(participants)-1 {
|
||||
resolveSession(session, "Consensus reached among participants")
|
||||
}
|
||||
```
|
||||
|
||||
**Session Resolved:**
|
||||
1. Update session status to "resolved"
|
||||
2. Record resolution reason
|
||||
3. Generate SLURP event (if integrator available)
|
||||
4. Broadcast resolution to participants
|
||||
5. Clean up after timeout
|
||||
|
||||
### 7. Session Escalation
|
||||
|
||||
**Escalation Triggers:**
|
||||
- Message count exceeds threshold (default: 10)
|
||||
- Session duration exceeds limit (default: 30 minutes)
|
||||
- Explicit escalation request from agent
|
||||
|
||||
**Escalation Process:**
|
||||
```go
|
||||
escalateSession(session, reason)
|
||||
↓
|
||||
Update status to "escalated"
|
||||
↓
|
||||
Generate SLURP event for human review
|
||||
↓
|
||||
Broadcast escalation notification
|
||||
↓
|
||||
Human intervention required
|
||||
```
|
||||
|
||||
## SLURP Integration
|
||||
|
||||
### Event Generation from Sessions
|
||||
|
||||
When sessions are resolved or escalated, the MetaCoordinator generates SLURP events:
|
||||
|
||||
```go
|
||||
discussionContext := integration.HmmmDiscussionContext{
|
||||
DiscussionID: session.SessionID,
|
||||
SessionID: session.SessionID,
|
||||
Participants: [agentIDs],
|
||||
StartTime: session.CreatedAt,
|
||||
EndTime: session.LastActivity,
|
||||
Messages: hmmmMessages,
|
||||
ConsensusReached: (outcome == "resolved"),
|
||||
ConsensusStrength: 0.9, // 0.3 for escalated, 0.5 for other
|
||||
OutcomeType: outcome, // "resolved" or "escalated"
|
||||
ProjectPath: projectPath,
|
||||
RelatedTasks: [taskIDs],
|
||||
Metadata: {
|
||||
"session_type": session.Type,
|
||||
"session_status": session.Status,
|
||||
"resolution": session.Resolution,
|
||||
"escalation_reason": session.EscalationReason,
|
||||
"message_count": len(session.Messages),
|
||||
"participant_count": len(session.Participants),
|
||||
},
|
||||
}
|
||||
|
||||
slurpIntegrator.ProcessHmmmDiscussion(ctx, discussionContext)
|
||||
```
|
||||
|
||||
**SLURP Event Outcomes:**
|
||||
- **Resolved sessions**: High consensus (0.9), successful coordination
|
||||
- **Escalated sessions**: Low consensus (0.3), human intervention needed
|
||||
- **Other outcomes**: Medium consensus (0.5)
|
||||
|
||||
### Policy Learning
|
||||
|
||||
SLURP uses coordination session data to learn:
|
||||
- Effective coordination patterns
|
||||
- Common dependency types
|
||||
- Escalation triggers
|
||||
- Agent collaboration efficiency
|
||||
- Task complexity indicators
|
||||
|
||||
## PubSub Message Types
|
||||
|
||||
### 1. dependency_detected
|
||||
|
||||
Announces a detected dependency between tasks.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "dependency_detected",
|
||||
"dependency": {
|
||||
"task1": {
|
||||
"task_id": 42,
|
||||
"project_id": 1,
|
||||
"repository": "backend-api",
|
||||
"title": "Define user authentication API",
|
||||
"agent_id": "agent-001"
|
||||
},
|
||||
"task2": {
|
||||
"task_id": 43,
|
||||
"project_id": 2,
|
||||
"repository": "frontend-app",
|
||||
"title": "Implement login page",
|
||||
"agent_id": "agent-002"
|
||||
},
|
||||
"relationship": "API_Contract",
|
||||
"confidence": 0.8,
|
||||
"reason": "API definition and implementation dependency",
|
||||
"detected_at": "2025-09-30T10:00:00Z"
|
||||
},
|
||||
"coordination_request": "Cross-repository dependency detected...",
|
||||
"agents_involved": ["agent-001", "agent-002"],
|
||||
"repositories": ["backend-api", "frontend-app"],
|
||||
"hop_count": 0,
|
||||
"max_hops": 3
|
||||
}
|
||||
```
|
||||
|
||||
### 2. coordination_plan
|
||||
|
||||
Broadcasts AI-generated coordination plan to participants.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "coordination_plan",
|
||||
"session_id": "dep_1_42_1727692800",
|
||||
"plan": "COORDINATION PLAN:\n1. SEQUENCE:\n...",
|
||||
"tasks_involved": [taskContext1, taskContext2],
|
||||
"participants": {
|
||||
"agent-001": { "agent_id": "agent-001", "repository": "backend-api" },
|
||||
"agent-002": { "agent_id": "agent-002", "repository": "frontend-app" }
|
||||
},
|
||||
"message": "Coordination plan generated for dependency: API_Contract"
|
||||
}
|
||||
```
|
||||
|
||||
### 3. coordination_response
|
||||
|
||||
Agent response to coordination plan or session message.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "coordination_response",
|
||||
"session_id": "dep_1_42_1727692800",
|
||||
"agent_id": "agent-001",
|
||||
"response": "I agree with the proposed sequence. API spec will be ready by EOD.",
|
||||
"timestamp": "2025-09-30T10:05:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### 4. session_message
|
||||
|
||||
General message within a coordination session.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "session_message",
|
||||
"session_id": "dep_1_42_1727692800",
|
||||
"from_agent": "agent-002",
|
||||
"content": "Can we schedule a quick sync to review the API spec?",
|
||||
"timestamp": "2025-09-30T10:10:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### 5. escalation
|
||||
|
||||
Session escalated to human intervention.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "escalation",
|
||||
"session_id": "dep_1_42_1727692800",
|
||||
"escalation_reason": "Message limit exceeded - human intervention needed",
|
||||
"session_summary": "Session dep_1_42_1727692800 (dependency): 2 participants, 12 messages, duration 35m",
|
||||
"participants": { /* participant info */ },
|
||||
"tasks_involved": [ /* task contexts */ ],
|
||||
"requires_human": true
|
||||
}
|
||||
```
|
||||
|
||||
### 6. resolution
|
||||
|
||||
Session successfully resolved.
|
||||
|
||||
```json
|
||||
{
|
||||
"message_type": "resolution",
|
||||
"session_id": "dep_1_42_1727692800",
|
||||
"resolution": "Consensus reached among participants",
|
||||
"summary": "Session dep_1_42_1727692800 (dependency): 2 participants, 8 messages, duration 15m"
|
||||
}
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Setup
|
||||
|
||||
```go
|
||||
import (
|
||||
"context"
|
||||
"chorus/pkg/coordination"
|
||||
"chorus/pubsub"
|
||||
)
|
||||
|
||||
// Create MetaCoordinator
|
||||
mc := coordination.NewMetaCoordinator(ctx, pubsubInstance)
|
||||
|
||||
// Optionally attach SLURP integrator
|
||||
mc.SetSlurpIntegrator(slurpIntegrator)
|
||||
|
||||
// MetaCoordinator automatically:
|
||||
// - Initializes DependencyDetector
|
||||
// - Sets up HMMM message handlers
|
||||
// - Starts session cleanup loop
|
||||
```
|
||||
|
||||
### Register Tasks for Dependency Detection
|
||||
|
||||
```go
|
||||
// Agent claims a task
|
||||
taskContext := &coordination.TaskContext{
|
||||
TaskID: 42,
|
||||
ProjectID: 1,
|
||||
Repository: "backend-api",
|
||||
Title: "Define user authentication API",
|
||||
Description: "Create OpenAPI spec for user auth endpoints",
|
||||
Keywords: []string{"api", "authentication", "openapi"},
|
||||
AgentID: "agent-001",
|
||||
ClaimedAt: time.Now(),
|
||||
}
|
||||
|
||||
mc.dependencyDetector.RegisterTask(taskContext)
|
||||
```
|
||||
|
||||
### Add Custom Dependency Rule
|
||||
|
||||
```go
|
||||
// Add project-specific rule
|
||||
microserviceRule := coordination.DependencyRule{
|
||||
Name: "Microservice_Interface",
|
||||
Description: "Microservice interface and consumer dependencies",
|
||||
Keywords: []string{"microservice", "interface", "consumer", "producer"},
|
||||
Validator: func(task1, task2 *coordination.TaskContext) (bool, string) {
|
||||
t1 := strings.ToLower(task1.Title + " " + task1.Description)
|
||||
t2 := strings.ToLower(task2.Title + " " + task2.Description)
|
||||
|
||||
hasProducer := strings.Contains(t1, "producer") || strings.Contains(t2, "producer")
|
||||
hasConsumer := strings.Contains(t1, "consumer") || strings.Contains(t2, "consumer")
|
||||
|
||||
if hasProducer && hasConsumer {
|
||||
return true, "Microservice producer and consumer must coordinate"
|
||||
}
|
||||
return false, ""
|
||||
},
|
||||
}
|
||||
|
||||
mc.dependencyDetector.AddCustomRule(microserviceRule)
|
||||
```
|
||||
|
||||
### Query Active Sessions
|
||||
|
||||
```go
|
||||
// Get all active coordination sessions
|
||||
sessions := mc.GetActiveSessions()
|
||||
|
||||
for sessionID, session := range sessions {
|
||||
fmt.Printf("Session %s:\n", sessionID)
|
||||
fmt.Printf(" Type: %s\n", session.Type)
|
||||
fmt.Printf(" Status: %s\n", session.Status)
|
||||
fmt.Printf(" Participants: %d\n", len(session.Participants))
|
||||
fmt.Printf(" Messages: %d\n", len(session.Messages))
|
||||
fmt.Printf(" Duration: %v\n", time.Since(session.CreatedAt))
|
||||
}
|
||||
```
|
||||
|
||||
### Monitor Coordination Events
|
||||
|
||||
```go
|
||||
// Set custom HMMM message handler
|
||||
pubsub.SetHmmmMessageHandler(func(msg pubsub.Message, from peer.ID) {
|
||||
switch msg.Data["message_type"] {
|
||||
case "dependency_detected":
|
||||
fmt.Printf("🔗 Dependency detected: %v\n", msg.Data)
|
||||
case "coordination_plan":
|
||||
fmt.Printf("📋 Coordination plan: %v\n", msg.Data)
|
||||
case "escalation":
|
||||
fmt.Printf("🚨 Escalation: %v\n", msg.Data)
|
||||
case "resolution":
|
||||
fmt.Printf("✅ Resolution: %v\n", msg.Data)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### MetaCoordinator Configuration
|
||||
|
||||
```go
|
||||
mc := coordination.NewMetaCoordinator(ctx, ps)
|
||||
|
||||
// Adjust session parameters
|
||||
mc.maxSessionDuration = 45 * time.Minute // Extend session timeout
|
||||
mc.maxParticipants = 10 // Support larger teams
|
||||
mc.escalationThreshold = 15 // More messages before escalation
|
||||
```
|
||||
|
||||
### DependencyDetector Configuration
|
||||
|
||||
```go
|
||||
dd := mc.dependencyDetector
|
||||
|
||||
// Adjust coordination hop limit
|
||||
dd.coordinationHops = 5 // Allow deeper meta-discussion chains
|
||||
```
|
||||
|
||||
## Session Lifecycle Management
|
||||
|
||||
### Automatic Cleanup
|
||||
|
||||
Sessions are automatically cleaned up by the session cleanup loop:
|
||||
|
||||
```go
|
||||
// Runs every 10 minutes
|
||||
func (mc *MetaCoordinator) cleanupInactiveSessions() {
|
||||
for sessionID, session := range mc.activeSessions {
|
||||
// Remove sessions older than 2 hours OR already resolved/escalated
|
||||
if time.Since(session.LastActivity) > 2*time.Hour ||
|
||||
session.Status == "resolved" ||
|
||||
session.Status == "escalated" {
|
||||
delete(mc.activeSessions, sessionID)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Cleanup Criteria:**
|
||||
- Session inactive for 2+ hours
|
||||
- Session status is "resolved"
|
||||
- Session status is "escalated"
|
||||
|
||||
### Manual Session Management
|
||||
|
||||
```go
|
||||
// Not exposed in current API, but could be added:
|
||||
|
||||
// Force resolve session
|
||||
mc.resolveSession(session, "Manual resolution by admin")
|
||||
|
||||
// Force escalate session
|
||||
mc.escalateSession(session, "Manual escalation requested")
|
||||
|
||||
// Cancel/close session
|
||||
mc.closeSession(sessionID)
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Memory Usage
|
||||
|
||||
- **TaskContext Storage**: ~500 bytes per task
|
||||
- **Active Sessions**: ~5KB per session (varies with message count)
|
||||
- **Dependency Rules**: ~1KB per rule
|
||||
|
||||
**Typical Usage**: 100 tasks + 10 sessions = ~100KB
|
||||
|
||||
### CPU Usage
|
||||
|
||||
- **Dependency Detection**: O(N²) where N = number of tasks per repository
|
||||
- **Rule Evaluation**: O(R) where R = number of rules
|
||||
- **Session Monitoring**: Periodic evaluation (every message received)
|
||||
|
||||
**Optimization**: Dependency detection skips same-repository comparisons.
|
||||
|
||||
### Network Usage
|
||||
|
||||
- **Dependency Announcements**: ~2KB per dependency
|
||||
- **Coordination Plans**: ~5KB per plan (includes full context)
|
||||
- **Session Messages**: ~1KB per message
|
||||
- **SLURP Events**: ~10KB per event (includes full session history)
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Rule Design
|
||||
|
||||
**Good Rule:**
|
||||
```go
|
||||
// Specific, actionable, clear success criteria
|
||||
{
|
||||
Name: "Database_Migration",
|
||||
Keywords: []string{"migration", "schema", "database"},
|
||||
Validator: func(t1, t2 *TaskContext) (bool, string) {
|
||||
// Clear matching logic
|
||||
// Specific reason returned
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**Bad Rule:**
|
||||
```go
|
||||
// Too broad, unclear coordination needed
|
||||
{
|
||||
Name: "Backend_Tasks",
|
||||
Keywords: []string{"backend"},
|
||||
Validator: func(t1, t2 *TaskContext) (bool, string) {
|
||||
return strings.Contains(t1.Title, "backend") &&
|
||||
strings.Contains(t2.Title, "backend"), "Both backend tasks"
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Session Participation
|
||||
|
||||
- **Respond promptly**: Keep sessions moving
|
||||
- **Be explicit**: Use clear agreement/disagreement language
|
||||
- **Stay focused**: Don't derail session with unrelated topics
|
||||
- **Escalate when stuck**: Don't let sessions drag on indefinitely
|
||||
|
||||
### 3. AI Plan Quality
|
||||
|
||||
AI plans are most effective when:
|
||||
- Task descriptions are detailed
|
||||
- Dependencies are clear
|
||||
- Agent capabilities are well-defined
|
||||
- Historical context is available
|
||||
|
||||
### 4. SLURP Integration
|
||||
|
||||
For best SLURP learning:
|
||||
- Enable SLURP integrator at startup
|
||||
- Ensure all sessions generate events (resolved or escalated)
|
||||
- Provide rich task metadata
|
||||
- Include project context in task descriptions
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Dependencies Not Detected
|
||||
|
||||
**Symptoms**: Related tasks not triggering coordination.
|
||||
|
||||
**Checks:**
|
||||
1. Verify tasks registered with detector: `dd.GetKnownTasks()`
|
||||
2. Check rule keywords match task content
|
||||
3. Test validator logic with task pairs
|
||||
4. Verify tasks are from different repositories
|
||||
5. Check PubSub connection for announcements
|
||||
|
||||
### Sessions Not Escalating
|
||||
|
||||
**Symptoms**: Long-running sessions without escalation.
|
||||
|
||||
**Checks:**
|
||||
1. Verify escalation threshold: `mc.escalationThreshold`
|
||||
2. Check session duration limit: `mc.maxSessionDuration`
|
||||
3. Verify message count in session
|
||||
4. Check for agreement keywords in messages
|
||||
5. Test escalation logic manually
|
||||
|
||||
### AI Plans Not Generated
|
||||
|
||||
**Symptoms**: Sessions created but no coordination plan.
|
||||
|
||||
**Checks:**
|
||||
1. Verify reasoning engine available: `reasoning.GenerateResponse()`
|
||||
2. Check AI model configuration
|
||||
3. Verify network connectivity to AI provider
|
||||
4. Check reasoning engine error logs
|
||||
5. Test with simpler dependency
|
||||
|
||||
### SLURP Events Not Generated
|
||||
|
||||
**Symptoms**: Sessions complete but no SLURP events.
|
||||
|
||||
**Checks:**
|
||||
1. Verify SLURP integrator attached: `mc.SetSlurpIntegrator()`
|
||||
2. Check SLURP integrator initialization
|
||||
3. Verify session outcome triggers event generation
|
||||
4. Check SLURP integrator error logs
|
||||
5. Test event generation manually
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
|
||||
1. **Machine Learning Rules**: Learn dependency patterns from historical data
|
||||
2. **Automated Testing**: Generate integration tests for coordinated tasks
|
||||
3. **Visualization**: Web UI for monitoring active sessions
|
||||
4. **Advanced Metrics**: Track coordination efficiency and success rates
|
||||
5. **Multi-Repo CI/CD**: Coordinate deployments across dependent services
|
||||
6. **Conflict Resolution**: AI-powered conflict resolution suggestions
|
||||
7. **Predictive Coordination**: Predict dependencies before tasks are claimed
|
||||
|
||||
## See Also
|
||||
|
||||
- [coordinator/](coordinator.md) - Task coordinator integration
|
||||
- [pubsub/](../pubsub.md) - PubSub messaging for coordination
|
||||
- [pkg/integration/](integration.md) - SLURP integration
|
||||
- [pkg/hmmm/](hmmm.md) - HMMM meta-discussion system
|
||||
- [reasoning/](../reasoning.md) - AI reasoning engine for planning
|
||||
- [internal/logging/](../internal/logging.md) - Hypercore logging
|
||||
Reference in New Issue
Block a user