CHORUS/docs/comprehensive/packages/slurp/README.md

# SLURP: Distributed Contextual Intelligence System

**Package:** `chorus/pkg/slurp`
**Status:** Production - Core System
**Complexity:** Very High - Multi-component distributed system

## Overview

SLURP (Storage, Logic, Understanding, Retrieval, Processing) is the contextual intelligence system for CHORUS, providing hierarchical context resolution, decision-based temporal analysis, distributed storage, and intelligent context generation across the cluster.

SLURP implements a sophisticated multi-layer architecture that tracks how code understanding evolves through decision points rather than just chronological time, enables role-based context sharing, and coordinates context generation through elected leader nodes.

## Architecture

### System Components

SLURP consists of eight integrated subpackages forming a comprehensive contextual intelligence platform:

```
pkg/slurp/
├── alignment/        # Goal alignment assessment and tracking
├── context/          # Hierarchical context resolution
├── distribution/     # Distributed context sharing via DHT
├── intelligence/     # AI-powered context generation
├── leader/           # Leader-based coordination
├── roles/            # Role-based access control
├── storage/          # Persistence and caching
└── temporal/         # Decision-hop temporal analysis
```

### Key Design Principles

1. **Decision-Hop Temporal Analysis**: Track context evolution by conceptual decision distance, not chronological time
2. **Bounded Hierarchy Traversal**: Prevent infinite loops while enabling cascading inheritance
3. **Leader-Only Generation**: Single elected leader generates context to prevent conflicts
4. **Role-Based Security**: Encrypt and filter context based on role permissions
5. **Distributed Coordination**: DHT-based storage with eventual consistency
6. **Multi-Layer Caching**: Local, distributed, and query caches for performance

### Component Relationships

```
┌─────────────────────────────────────────────────────────────────┐
│                          SLURP Core                              │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │              Main SLURP Coordinator                        │  │
│  │  • Context Resolution Orchestration                        │  │
│  │  • Temporal Graph Management                               │  │
│  │  • Storage Coordination                                    │  │
│  │  • Event System                                            │  │
│  └──────┬─────────────┬───────────────┬─────────────┬────────┘  │
│         │             │               │             │            │
│    ┌────▼────┐   ┌───▼────┐     ┌────▼────┐  ┌────▼────┐       │
│    │Context  │   │Temporal│     │Storage  │  │Leader   │       │
│    │Resolver │   │Graph   │     │Layer    │  │Manager  │       │
│    └────┬────┘   └───┬────┘     └────┬────┘  └────┬────┘       │
│         │            │               │            │             │
└─────────┼────────────┼───────────────┼────────────┼─────────────┘
          │            │               │            │
     ┌────▼────┐  ┌───▼────┐     ┌────▼────┐  ┌────▼────┐
     │Alignment│  │Intelli-│     │Distri-  │  │Roles    │
     │Analyzer │  │gence   │     │bution   │  │Manager  │
     └─────────┘  └────────┘     └─────────┘  └─────────┘
          │            │               │            │
          └────────────┴───────────────┴────────────┘
                       │
            Integration with CHORUS Systems:
            • pkg/dht - Distributed storage
            • pkg/election - Leader coordination
            • pkg/crypto - Role-based encryption
            • pkg/ucxl - Address resolution
```

## Core Functionality

### 1. Hierarchical Context Resolution

Resolves context for UCXL addresses using cascading inheritance similar to CSS:

```go
// Resolve context with bounded depth traversal
resolved, err := slurp.Resolve(ctx, "ucxl://chorus/pkg/slurp/context/resolver.go")
if err != nil {
    return err
}

fmt.Printf("Summary: %s\n", resolved.Summary)
fmt.Printf("Technologies: %v\n", resolved.Technologies)
fmt.Printf("Inheritance chain: %v\n", resolved.InheritanceChain)
fmt.Printf("Bounded depth: %d\n", resolved.BoundedDepth)
```

**Features:**
- Bounded hierarchy traversal (prevents infinite loops)
- CSS-like cascading and inheritance
- Multi-level caching with TTL
- Role-based filtering of results
- Global context application

### 2. Decision-Hop Temporal Analysis

Track context evolution through decision influence graphs:

```go
// Get temporal evolution history
history, err := slurp.GetTemporalEvolution(ctx, address)
for _, node := range history {
    fmt.Printf("Version %d: %s (Decision: %s)\n",
               node.Version, node.ChangeReason, node.DecisionID)
}

// Navigate by decision hops, not time
threeHopsBack, err := slurp.NavigateDecisionHops(ctx, address, 3, NavigationBackward)
```

**Features:**
- Decision-hop distance instead of chronological time
- Influence graph tracking which decisions affect others
- Decision timeline reconstruction
- Staleness detection based on decision relationships
- Pattern analysis in decision-making

### 3. Context Generation (Leader-Only)

Intelligent context generation restricted to elected admin nodes:

```go
// Check if current node is admin
if slurp.IsCurrentNodeAdmin() {
    options := &GenerationOptions{
        AnalyzeContent:      true,
        AnalyzeStructure:    true,
        AnalyzeHistory:      true,
        UseRAG:              true,
        EncryptForRoles:     []string{"developer", "architect"},
    }

    generated, err := slurp.GenerateContext(ctx, "/path/to/code", options)
    if err != nil {
        return err
    }
}
```

**Features:**
- Admin-only restriction prevents conflicts
- Multi-source analysis (content, structure, history)
- RAG system integration for enhanced understanding
- Quality validation and confidence scoring
- Role-based encryption of generated context

### 4. Distributed Storage and Coordination

DHT-based distributed context sharing:

```go
// Context automatically stored and replicated across cluster
context, err := slurp.UpsertContext(ctx, contextNode)

// Batch resolution with distributed cache
addresses := []string{
    "ucxl://chorus/pkg/dht/...",
    "ucxl://chorus/pkg/election/...",
}
results, err := slurp.BatchResolve(ctx, addresses)
```

**Features:**
- DHT-based distributed storage
- Role-based encryption for secure sharing
- Configurable replication factors
- Eventual consistency with conflict resolution
- Network partition resilience

### 5. Role-Based Access Control

Comprehensive RBAC for context information:

```go
// Context filtered and encrypted based on role
resolved, err := slurp.Resolve(ctx, address)
// Returns only information accessible to current role

// Different roles see different context perspectives
// - Developers: Implementation details, code patterns
// - Architects: Design decisions, structural information
// - Product: Business alignment, goal tracking
```

**Features:**
- Hierarchical role definitions
- Multi-role context encryption
- Dynamic permission evaluation
- Audit logging of access decisions
- Temporal access control (time-limited permissions)

## Configuration

### Basic Configuration

```yaml
slurp:
  enabled: true

  # Context resolution settings
  context_resolution:
    max_hierarchy_depth: 10
    default_depth_limit: 5
    cache_ttl: 15m
    cache_max_entries: 1000
    min_confidence_threshold: 0.6
    enable_global_contexts: true

  # Temporal analysis settings
  temporal_analysis:
    max_decision_hops: 10
    default_hop_limit: 5
    enable_navigation: true
    staleness_threshold: 0.2
    staleness_check_interval: 5m
    enable_influence_propagation: true

  # Storage configuration
  storage:
    backend: "hybrid"  # dht or hybrid
    default_encryption: true
    encryption_roles: ["developer", "architect", "admin"]
    local_cache_enabled: true
    local_cache_path: "/home/user/.chorus/slurp"
    sync_interval: 30s
    replication_factor: 3
    consistency_level: "eventual"

  # Intelligence/generation settings (admin-only)
  intelligence:
    enable_generation: true
    generation_timeout: 5m
    generation_concurrency: 4
    enable_analysis: true
    enable_pattern_detection: true
    pattern_match_threshold: 0.75
    rag_endpoint: "http://localhost:8080"

  # Performance tuning
  performance:
    max_concurrent_resolutions: 50
    max_concurrent_generations: 4
    default_request_timeout: 30s
    background_task_timeout: 10m
    enable_metrics: true
    metrics_collection_interval: 1m

  # Security settings
  security:
    enforce_role_based_access: true
    default_access_roles: ["developer"]
    admin_only_operations:
      - "generate_context"
      - "regenerate_hierarchy"
      - "modify_global_context"
    enable_audit_log: true
    require_encryption: true
```

### Advanced Configuration

```yaml
slurp:
  # Advanced context resolution
  context_resolution:
    require_strict_matching: false
    allow_partial_resolution: true
    global_context_ttl: 1h

  # Advanced temporal settings
  temporal_analysis:
    max_navigation_history: 100
    min_decision_confidence: 0.5
    max_decision_age: 90d
    max_influence_depth: 5

  # Advanced storage
  storage:
    local_cache_max_size: 1GB
    sync_timeout: 10s
    conflict_resolution: "last_writer_wins"

  # Quality settings
  intelligence:
    quality_threshold: 0.7
    enable_quality_metrics: true
    rag_timeout: 10s

  # Resource limits
  performance:
    max_memory_usage: 2GB
    max_disk_usage: 10GB
    default_batch_size: 10
    max_batch_size: 100
    batch_timeout: 1m

  # Advanced security
  security:
    audit_log_path: "/var/log/chorus/slurp-audit.log"
    log_sensitive_operations: true
    encryption_algorithm: "age"
    key_rotation_interval: 30d
    enable_rate_limiting: true
    default_rate_limit: 100
    burst_limit: 200
```

## Usage Patterns

### Pattern 1: Basic Context Resolution

```go
// Create SLURP instance
slurp, err := slurp.NewSLURP(config, dht, crypto, election)
if err != nil {
    return err
}

// Initialize system
if err := slurp.Initialize(ctx); err != nil {
    return err
}
defer slurp.Close()

// Resolve context
resolved, err := slurp.Resolve(ctx, "ucxl://project/src/main.go")
if err != nil {
    return err
}

fmt.Printf("Context: %s\n", resolved.Summary)
```

### Pattern 2: Temporal Navigation

```go
// Get evolution history
history, err := slurp.GetTemporalEvolution(ctx, address)
for _, node := range history {
    fmt.Printf("Version %d at %s: %s\n",
               node.Version, node.Timestamp, node.ChangeReason)
}

// Navigate decision graph
navigator := temporal.NewNavigator(slurp.temporalGraph)
timeline, err := navigator.GetDecisionTimeline(ctx, address, true, 5)

fmt.Printf("Total decisions: %d\n", timeline.TotalDecisions)
for _, entry := range timeline.DecisionSequence {
    fmt.Printf("Hop %d: %s by %s\n",
               entry.DecisionHop, entry.ChangeReason, entry.DecisionMaker)
}
```

### Pattern 3: Leader-Based Context Generation

```go
// Check leadership status
if !slurp.IsCurrentNodeAdmin() {
    return fmt.Errorf("context generation requires admin role")
}

// Generate context with analysis
options := &GenerationOptions{
    AnalyzeContent:      true,
    AnalyzeStructure:    true,
    AnalyzeHistory:      true,
    AnalyzeDependencies: true,
    UseRAG:              true,
    MaxDepth:            3,
    MinConfidence:       0.7,
    EncryptForRoles:     []string{"developer", "architect"},
}

generated, err := slurp.GenerateContext(ctx, "/project/src", options)
if err != nil {
    return err
}

fmt.Printf("Generated context with confidence: %.2f\n", generated.Confidence)
```

### Pattern 4: Batch Resolution for Performance

```go
// Batch resolve multiple addresses efficiently
addresses := []string{
    "ucxl://project/src/api/handler.go",
    "ucxl://project/src/api/middleware.go",
    "ucxl://project/src/api/router.go",
}

results, err := slurp.BatchResolve(ctx, addresses)
if err != nil {
    return err
}

for addr, resolved := range results {
    fmt.Printf("%s: %s\n", addr, resolved.Summary)
}
```

### Pattern 5: Event Handling

```go
// Register event handlers for monitoring
slurp.RegisterEventHandler(EventContextGenerated, func(ctx context.Context, event *SLURPEvent) error {
    fmt.Printf("Context generated: %v\n", event.Data)
    return nil
})

slurp.RegisterEventHandler(EventAdminChanged, func(ctx context.Context, event *SLURPEvent) error {
    fmt.Printf("Admin changed: %s -> %s\n",
               event.Data["old_admin"], event.Data["new_admin"])
    return nil
})

slurp.RegisterEventHandler(EventStalenessDetected, func(ctx context.Context, event *SLURPEvent) error {
    fmt.Printf("Stale context detected: %v\n", event.Data)
    return nil
})
```

## Integration with CHORUS Systems

### Election System Integration

```go
// SLURP automatically integrates with election system
// Admin status updated on election changes
election.SetCallbacks(
    slurp.handleAdminChanged,
    slurp.handleElectionComplete,
)

// Context generation restricted to admin
if slurp.IsCurrentNodeAdmin() {
    // Only admin can generate context
    generated, err := slurp.GenerateContext(ctx, path, options)
}
```

### DHT Integration

```go
// SLURP uses DHT for distributed storage
// Contexts automatically replicated across cluster
contextData := slurp.Resolve(ctx, address)
// Data retrieved from local cache or DHT as needed

// Storage layer handles DHT operations transparently
slurp.UpsertContext(ctx, contextNode)
// Automatically stored locally and replicated to DHT
```

### Crypto Integration

```go
// Role-based encryption handled automatically
context := &ContextNode{
    // ...
    EncryptedFor: []string{"developer", "architect"},
    AccessLevel:  crypto.AccessLevelHigh,
}

// Context encrypted before storage
// Only authorized roles can decrypt
slurp.UpsertContext(ctx, context)
```

### UCXL Integration

```go
// SLURP understands UCXL addresses natively
address := "ucxl://project/src/api/handler.go"
resolved, err := slurp.Resolve(ctx, address)

// Handles full UCXL syntax including:
// - Hierarchical paths
// - Query parameters
// - Fragments
// - Version specifiers
```

## Performance Characteristics

### Resolution Performance

- **Cache Hit**: < 1ms (in-memory cache)
- **Cache Miss (Local Storage)**: 5-10ms (LevelDB lookup)
- **Cache Miss (DHT)**: 50-200ms (network + DHT lookup)
- **Hierarchy Traversal**: O(depth) with typical depth 3-5 levels
- **Batch Resolution**: 10-100x faster than sequential for large batches

### Storage Performance

- **Local Write**: 1-5ms (LevelDB)
- **Distributed Write**: 50-200ms (DHT replication)
- **Sync Operation**: 100-500ms (cluster-wide)
- **Index Build**: O(N log N) with background optimization
- **Query Performance**: 10-100ms with indexes

### Temporal Analysis Performance

- **Decision Path Query**: 10-50ms (graph traversal)
- **Evolution History**: 5-20ms (indexed lookup)
- **Staleness Detection**: Background task, no user impact
- **Navigation**: O(hops) with typical 3-10 hops
- **Influence Analysis**: 50-200ms (graph analysis)

### Memory Usage

- **Base System**: ~50MB
- **Cache (per 1000 contexts)**: ~100MB
- **Temporal Graph**: ~20MB per 1000 nodes
- **Index Structures**: ~50MB per 10000 contexts
- **Total Typical**: 200-500MB for medium project

## Monitoring and Metrics

### Key Metrics

```go
metrics := slurp.GetMetrics()

// Resolution metrics
fmt.Printf("Total resolutions: %d\n", metrics.TotalResolutions)
fmt.Printf("Success rate: %.2f%%\n",
           float64(metrics.SuccessfulResolutions)/float64(metrics.TotalResolutions)*100)
fmt.Printf("Cache hit rate: %.2f%%\n", metrics.CacheHitRate*100)
fmt.Printf("Average resolution time: %v\n", metrics.AverageResolutionTime)

// Temporal metrics
fmt.Printf("Temporal nodes: %d\n", metrics.TemporalNodes)
fmt.Printf("Decision paths: %d\n", metrics.DecisionPaths)
fmt.Printf("Stale contexts: %d\n", metrics.StaleContexts)

// Storage metrics
fmt.Printf("Stored contexts: %d\n", metrics.StoredContexts)
fmt.Printf("Encrypted contexts: %d\n", metrics.EncryptedContexts)
fmt.Printf("Storage utilization: %.2f%%\n", metrics.StorageUtilization*100)

// Intelligence metrics
fmt.Printf("Generation requests: %d\n", metrics.GenerationRequests)
fmt.Printf("Successful generations: %d\n", metrics.SuccessfulGenerations)
fmt.Printf("Pattern matches: %d\n", metrics.PatternMatches)
```

### Event Monitoring

```go
// Monitor system events
slurp.RegisterEventHandler(EventContextResolved, metricsCollector)
slurp.RegisterEventHandler(EventContextGenerated, auditLogger)
slurp.RegisterEventHandler(EventErrorOccurred, errorTracker)
slurp.RegisterEventHandler(EventStalenessDetected, alertSystem)
```

## Implementation Status

### Completed Features

- **Core SLURP Coordinator**: Production-ready main coordinator
- **Context Resolution**: Bounded hierarchy traversal with caching
- **Temporal Graph**: Decision-hop temporal analysis fully implemented
- **Storage Layer**: Local and distributed storage operational
- **Leader Integration**: Election-based leader coordination working
- **Role-Based Security**: Encryption and access control functional
- **Event System**: Event handling and notification working
- **Metrics Collection**: Performance monitoring active

### In Development

- **Alignment Analyzer**: Goal alignment assessment (stubs in place)
- **Intelligence Engine**: Context generation engine (partial implementation)
- **Distribution Layer**: Full DHT-based distribution (partial)
- **Pattern Detection**: Advanced pattern matching capabilities
- **Query Optimization**: Advanced query and search features

### Experimental Features

- **RAG Integration**: External RAG system integration (experimental)
- **Multi-language Analysis**: Beyond Go language support
- **Graph Visualization**: Temporal graph visualization tools
- **ML-Based Staleness**: Machine learning for staleness prediction
- **Automated Repair**: Self-healing context inconsistencies

## Troubleshooting

### Common Issues

#### Issue: Context Not Found

```go
// Symptom
resolved, err := slurp.Resolve(ctx, address)
// Returns: "context not found for ucxl://..."

// Causes:
// 1. Context never generated for this address
// 2. Cache invalidated and persistence not enabled
// 3. Role permissions prevent access

// Solutions:
// 1. Generate context (if admin)
if slurp.IsCurrentNodeAdmin() {
    generated, err := slurp.GenerateContext(ctx, path, options)
}

// 2. Check role permissions
// 3. Verify storage configuration
```

#### Issue: High Resolution Latency

```go
// Symptom: Slow context resolution (> 1 second)

// Causes:
// 1. Cache disabled or not warming up
// 2. Deep hierarchy traversal
// 3. Network issues with DHT
// 4. Storage backend slow

// Solutions:
// 1. Enable caching with appropriate TTL
config.Slurp.ContextResolution.CacheTTL = 15 * time.Minute

// 2. Reduce depth limit
resolved, err := slurp.ResolveWithDepth(ctx, address, 3)

// 3. Use batch resolution
results, err := slurp.BatchResolve(ctx, addresses)

// 4. Check storage metrics
metrics := slurp.GetMetrics()
fmt.Printf("Cache hit rate: %.2f%%\n", metrics.CacheHitRate*100)
```

#### Issue: Admin Node Not Generating Context

```go
// Symptom: Context generation fails with "requires admin privileges"

// Causes:
// 1. Node not elected as admin
// 2. Election system not initialized
// 3. Leadership change in progress

// Solutions:
// 1. Check admin status
if !slurp.IsCurrentNodeAdmin() {
    fmt.Printf("Current admin: %s\n", slurp.currentAdmin)
    // Wait for election or request from admin
}

// 2. Verify election system
if election.GetCurrentAdmin() == "" {
    // No admin elected yet
}

// 3. Monitor admin changes
slurp.RegisterEventHandler(EventAdminChanged, handler)
```

#### Issue: Temporal Navigation Returns No Results

```go
// Symptom: GetTemporalEvolution returns empty array

// Causes:
// 1. Temporal tracking not enabled
// 2. No evolution recorded for this context
// 3. Temporal storage not initialized

// Solutions:
// 1. Evolve context when changes occur
decision := &DecisionMetadata{/*...*/}
evolved, err := slurp.temporalGraph.EvolveContext(ctx, address, newContext, reason, decision)

// 2. Check temporal system initialization
if slurp.temporalGraph == nil {
    // Temporal system not initialized
}

// 3. Verify temporal storage
if slurp.temporalStore == nil {
    // Storage not configured
}
```

## Related Packages

- **pkg/dht**: Distributed Hash Table for storage
- **pkg/election**: Leader election for coordination
- **pkg/crypto**: Role-based encryption and access control
- **pkg/ucxl**: UCXL address parsing and handling
- **pkg/config**: Configuration management

## Subpackage Documentation

Detailed documentation for each subpackage:

- [alignment/](./alignment.md) - Goal alignment assessment and tracking
- [context/](./context.md) - Hierarchical context resolution
- [distribution/](./distribution.md) - Distributed context sharing
- [intelligence/](./intelligence.md) - AI-powered context generation
- [leader/](./leader.md) - Leader-based coordination
- [roles/](./roles.md) - Role-based access control
- [storage/](./storage.md) - Persistence and caching layer
- [temporal/](./temporal.md) - Decision-hop temporal analysis

## Further Reading

- CHORUS Architecture Documentation
- DHT Design and Implementation
- Election System Documentation
- Role-Based Access Control Guide
- UCXL Address Specification