CHORUS/docs/comprehensive/packages/slurp

SLURP: Distributed Contextual Intelligence System

Package: chorus/pkg/slurp Status: Production - Core System Complexity: Very High - Multi-component distributed system

Overview

SLURP (Storage, Logic, Understanding, Retrieval, Processing) is the contextual intelligence system for CHORUS, providing hierarchical context resolution, decision-based temporal analysis, distributed storage, and intelligent context generation across the cluster.

SLURP implements a sophisticated multi-layer architecture that tracks how code understanding evolves through decision points rather than just chronological time, enables role-based context sharing, and coordinates context generation through elected leader nodes.

Architecture

System Components

SLURP consists of eight integrated subpackages forming a comprehensive contextual intelligence platform:

pkg/slurp/
├── alignment/        # Goal alignment assessment and tracking
├── context/          # Hierarchical context resolution
├── distribution/     # Distributed context sharing via DHT
├── intelligence/     # AI-powered context generation
├── leader/           # Leader-based coordination
├── roles/            # Role-based access control
├── storage/          # Persistence and caching
└── temporal/         # Decision-hop temporal analysis

Key Design Principles

1. Decision-Hop Temporal Analysis: Track context evolution by conceptual decision distance, not chronological time
2. Bounded Hierarchy Traversal: Prevent infinite loops while enabling cascading inheritance
3. Leader-Only Generation: Single elected leader generates context to prevent conflicts
4. Role-Based Security: Encrypt and filter context based on role permissions
5. Distributed Coordination: DHT-based storage with eventual consistency
6. Multi-Layer Caching: Local, distributed, and query caches for performance

Component Relationships

┌─────────────────────────────────────────────────────────────────┐
│                          SLURP Core                              │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │              Main SLURP Coordinator                        │  │
│  │  • Context Resolution Orchestration                        │  │
│  │  • Temporal Graph Management                               │  │
│  │  • Storage Coordination                                    │  │
│  │  • Event System                                            │  │
│  └──────┬─────────────┬───────────────┬─────────────┬────────┘  │
│         │             │               │             │            │
│    ┌────▼────┐   ┌───▼────┐     ┌────▼────┐  ┌────▼────┐       │
│    │Context  │   │Temporal│     │Storage  │  │Leader   │       │
│    │Resolver │   │Graph   │     │Layer    │  │Manager  │       │
│    └────┬────┘   └───┬────┘     └────┬────┘  └────┬────┘       │
│         │            │               │            │             │
└─────────┼────────────┼───────────────┼────────────┼─────────────┘
          │            │               │            │
     ┌────▼────┐  ┌───▼────┐     ┌────▼────┐  ┌────▼────┐
     │Alignment│  │Intelli-│     │Distri-  │  │Roles    │
     │Analyzer │  │gence   │     │bution   │  │Manager  │
     └─────────┘  └────────┘     └─────────┘  └─────────┘
          │            │               │            │
          └────────────┴───────────────┴────────────┘
                       │
            Integration with CHORUS Systems:
            • pkg/dht - Distributed storage
            • pkg/election - Leader coordination
            • pkg/crypto - Role-based encryption
            • pkg/ucxl - Address resolution

Core Functionality

1. Hierarchical Context Resolution

Resolves context for UCXL addresses using cascading inheritance similar to CSS:

// Resolve context with bounded depth traversal
resolved, err := slurp.Resolve(ctx, "ucxl://chorus/pkg/slurp/context/resolver.go")
if err != nil {
    return err
}

fmt.Printf("Summary: %s\n", resolved.Summary)
fmt.Printf("Technologies: %v\n", resolved.Technologies)
fmt.Printf("Inheritance chain: %v\n", resolved.InheritanceChain)
fmt.Printf("Bounded depth: %d\n", resolved.BoundedDepth)

Features:

• Bounded hierarchy traversal (prevents infinite loops)
• CSS-like cascading and inheritance
• Multi-level caching with TTL
• Role-based filtering of results
• Global context application

2. Decision-Hop Temporal Analysis

Track context evolution through decision influence graphs:

// Get temporal evolution history
history, err := slurp.GetTemporalEvolution(ctx, address)
for _, node := range history {
    fmt.Printf("Version %d: %s (Decision: %s)\n",
               node.Version, node.ChangeReason, node.DecisionID)
}

// Navigate by decision hops, not time
threeHopsBack, err := slurp.NavigateDecisionHops(ctx, address, 3, NavigationBackward)

Features:

• Decision-hop distance instead of chronological time
• Influence graph tracking which decisions affect others
• Decision timeline reconstruction
• Staleness detection based on decision relationships
• Pattern analysis in decision-making

3. Context Generation (Leader-Only)

Intelligent context generation restricted to elected admin nodes:

// Check if current node is admin
if slurp.IsCurrentNodeAdmin() {
    options := &GenerationOptions{
        AnalyzeContent:      true,
        AnalyzeStructure:    true,
        AnalyzeHistory:      true,
        UseRAG:              true,
        EncryptForRoles:     []string{"developer", "architect"},
    }

    generated, err := slurp.GenerateContext(ctx, "/path/to/code", options)
    if err != nil {
        return err
    }
}

Features:

• Admin-only restriction prevents conflicts
• Multi-source analysis (content, structure, history)
• RAG system integration for enhanced understanding
• Quality validation and confidence scoring
• Role-based encryption of generated context

4. Distributed Storage and Coordination

DHT-based distributed context sharing:

// Context automatically stored and replicated across cluster
context, err := slurp.UpsertContext(ctx, contextNode)

// Batch resolution with distributed cache
addresses := []string{
    "ucxl://chorus/pkg/dht/...",
    "ucxl://chorus/pkg/election/...",
}
results, err := slurp.BatchResolve(ctx, addresses)

Features:

• DHT-based distributed storage
• Role-based encryption for secure sharing
• Configurable replication factors
• Eventual consistency with conflict resolution
• Network partition resilience

5. Role-Based Access Control

Comprehensive RBAC for context information:

// Context filtered and encrypted based on role
resolved, err := slurp.Resolve(ctx, address)
// Returns only information accessible to current role

// Different roles see different context perspectives
// - Developers: Implementation details, code patterns
// - Architects: Design decisions, structural information
// - Product: Business alignment, goal tracking

Features:

• Hierarchical role definitions
• Multi-role context encryption
• Dynamic permission evaluation
• Audit logging of access decisions
• Temporal access control (time-limited permissions)

Configuration

Basic Configuration

slurp:
  enabled: true

  # Context resolution settings
  context_resolution:
    max_hierarchy_depth: 10
    default_depth_limit: 5
    cache_ttl: 15m
    cache_max_entries: 1000
    min_confidence_threshold: 0.6
    enable_global_contexts: true

  # Temporal analysis settings
  temporal_analysis:
    max_decision_hops: 10
    default_hop_limit: 5
    enable_navigation: true
    staleness_threshold: 0.2
    staleness_check_interval: 5m
    enable_influence_propagation: true

  # Storage configuration
  storage:
    backend: "hybrid"  # dht or hybrid
    default_encryption: true
    encryption_roles: ["developer", "architect", "admin"]
    local_cache_enabled: true
    local_cache_path: "/home/user/.chorus/slurp"
    sync_interval: 30s
    replication_factor: 3
    consistency_level: "eventual"

  # Intelligence/generation settings (admin-only)
  intelligence:
    enable_generation: true
    generation_timeout: 5m
    generation_concurrency: 4
    enable_analysis: true
    enable_pattern_detection: true
    pattern_match_threshold: 0.75
    rag_endpoint: "http://localhost:8080"

  # Performance tuning
  performance:
    max_concurrent_resolutions: 50
    max_concurrent_generations: 4
    default_request_timeout: 30s
    background_task_timeout: 10m
    enable_metrics: true
    metrics_collection_interval: 1m

  # Security settings
  security:
    enforce_role_based_access: true
    default_access_roles: ["developer"]
    admin_only_operations:
      - "generate_context"
      - "regenerate_hierarchy"
      - "modify_global_context"
    enable_audit_log: true
    require_encryption: true

Advanced Configuration

slurp:
  # Advanced context resolution
  context_resolution:
    require_strict_matching: false
    allow_partial_resolution: true
    global_context_ttl: 1h

  # Advanced temporal settings
  temporal_analysis:
    max_navigation_history: 100
    min_decision_confidence: 0.5
    max_decision_age: 90d
    max_influence_depth: 5

  # Advanced storage
  storage:
    local_cache_max_size: 1GB
    sync_timeout: 10s
    conflict_resolution: "last_writer_wins"

  # Quality settings
  intelligence:
    quality_threshold: 0.7
    enable_quality_metrics: true
    rag_timeout: 10s

  # Resource limits
  performance:
    max_memory_usage: 2GB
    max_disk_usage: 10GB
    default_batch_size: 10
    max_batch_size: 100
    batch_timeout: 1m

  # Advanced security
  security:
    audit_log_path: "/var/log/chorus/slurp-audit.log"
    log_sensitive_operations: true
    encryption_algorithm: "age"
    key_rotation_interval: 30d
    enable_rate_limiting: true
    default_rate_limit: 100
    burst_limit: 200

Usage Patterns

Pattern 1: Basic Context Resolution

// Create SLURP instance
slurp, err := slurp.NewSLURP(config, dht, crypto, election)
if err != nil {
    return err
}

// Initialize system
if err := slurp.Initialize(ctx); err != nil {
    return err
}
defer slurp.Close()

// Resolve context
resolved, err := slurp.Resolve(ctx, "ucxl://project/src/main.go")
if err != nil {
    return err
}

fmt.Printf("Context: %s\n", resolved.Summary)

Pattern 2: Temporal Navigation

// Get evolution history
history, err := slurp.GetTemporalEvolution(ctx, address)
for _, node := range history {
    fmt.Printf("Version %d at %s: %s\n",
               node.Version, node.Timestamp, node.ChangeReason)
}

// Navigate decision graph
navigator := temporal.NewNavigator(slurp.temporalGraph)
timeline, err := navigator.GetDecisionTimeline(ctx, address, true, 5)

fmt.Printf("Total decisions: %d\n", timeline.TotalDecisions)
for _, entry := range timeline.DecisionSequence {
    fmt.Printf("Hop %d: %s by %s\n",
               entry.DecisionHop, entry.ChangeReason, entry.DecisionMaker)
}

Pattern 3: Leader-Based Context Generation

// Check leadership status
if !slurp.IsCurrentNodeAdmin() {
    return fmt.Errorf("context generation requires admin role")
}

// Generate context with analysis
options := &GenerationOptions{
    AnalyzeContent:      true,
    AnalyzeStructure:    true,
    AnalyzeHistory:      true,
    AnalyzeDependencies: true,
    UseRAG:              true,
    MaxDepth:            3,
    MinConfidence:       0.7,
    EncryptForRoles:     []string{"developer", "architect"},
}

generated, err := slurp.GenerateContext(ctx, "/project/src", options)
if err != nil {
    return err
}

fmt.Printf("Generated context with confidence: %.2f\n", generated.Confidence)

Pattern 4: Batch Resolution for Performance

// Batch resolve multiple addresses efficiently
addresses := []string{
    "ucxl://project/src/api/handler.go",
    "ucxl://project/src/api/middleware.go",
    "ucxl://project/src/api/router.go",
}

results, err := slurp.BatchResolve(ctx, addresses)
if err != nil {
    return err
}

for addr, resolved := range results {
    fmt.Printf("%s: %s\n", addr, resolved.Summary)
}

Pattern 5: Event Handling

// Register event handlers for monitoring
slurp.RegisterEventHandler(EventContextGenerated, func(ctx context.Context, event *SLURPEvent) error {
    fmt.Printf("Context generated: %v\n", event.Data)
    return nil
})

slurp.RegisterEventHandler(EventAdminChanged, func(ctx context.Context, event *SLURPEvent) error {
    fmt.Printf("Admin changed: %s -> %s\n",
               event.Data["old_admin"], event.Data["new_admin"])
    return nil
})

slurp.RegisterEventHandler(EventStalenessDetected, func(ctx context.Context, event *SLURPEvent) error {
    fmt.Printf("Stale context detected: %v\n", event.Data)
    return nil
})

Integration with CHORUS Systems

Election System Integration

// SLURP automatically integrates with election system
// Admin status updated on election changes
election.SetCallbacks(
    slurp.handleAdminChanged,
    slurp.handleElectionComplete,
)

// Context generation restricted to admin
if slurp.IsCurrentNodeAdmin() {
    // Only admin can generate context
    generated, err := slurp.GenerateContext(ctx, path, options)
}

DHT Integration

// SLURP uses DHT for distributed storage
// Contexts automatically replicated across cluster
contextData := slurp.Resolve(ctx, address)
// Data retrieved from local cache or DHT as needed

// Storage layer handles DHT operations transparently
slurp.UpsertContext(ctx, contextNode)
// Automatically stored locally and replicated to DHT

Crypto Integration

// Role-based encryption handled automatically
context := &ContextNode{
    // ...
    EncryptedFor: []string{"developer", "architect"},
    AccessLevel:  crypto.AccessLevelHigh,
}

// Context encrypted before storage
// Only authorized roles can decrypt
slurp.UpsertContext(ctx, context)

UCXL Integration

// SLURP understands UCXL addresses natively
address := "ucxl://project/src/api/handler.go"
resolved, err := slurp.Resolve(ctx, address)

// Handles full UCXL syntax including:
// - Hierarchical paths
// - Query parameters
// - Fragments
// - Version specifiers

Performance Characteristics

Resolution Performance

• Cache Hit: < 1ms (in-memory cache)
• Cache Miss (Local Storage): 5-10ms (LevelDB lookup)
• Cache Miss (DHT): 50-200ms (network + DHT lookup)
• Hierarchy Traversal: O(depth) with typical depth 3-5 levels
• Batch Resolution: 10-100x faster than sequential for large batches

Storage Performance

• Local Write: 1-5ms (LevelDB)
• Distributed Write: 50-200ms (DHT replication)
• Sync Operation: 100-500ms (cluster-wide)
• Index Build: O(N log N) with background optimization
• Query Performance: 10-100ms with indexes

Temporal Analysis Performance

• Decision Path Query: 10-50ms (graph traversal)
• Evolution History: 5-20ms (indexed lookup)
• Staleness Detection: Background task, no user impact
• Navigation: O(hops) with typical 3-10 hops
• Influence Analysis: 50-200ms (graph analysis)

Memory Usage

• Base System: ~50MB
• Cache (per 1000 contexts): ~100MB
• Temporal Graph: ~20MB per 1000 nodes
• Index Structures: ~50MB per 10000 contexts
• Total Typical: 200-500MB for medium project

Monitoring and Metrics

Key Metrics

metrics := slurp.GetMetrics()

// Resolution metrics
fmt.Printf("Total resolutions: %d\n", metrics.TotalResolutions)
fmt.Printf("Success rate: %.2f%%\n",
           float64(metrics.SuccessfulResolutions)/float64(metrics.TotalResolutions)*100)
fmt.Printf("Cache hit rate: %.2f%%\n", metrics.CacheHitRate*100)
fmt.Printf("Average resolution time: %v\n", metrics.AverageResolutionTime)

// Temporal metrics
fmt.Printf("Temporal nodes: %d\n", metrics.TemporalNodes)
fmt.Printf("Decision paths: %d\n", metrics.DecisionPaths)
fmt.Printf("Stale contexts: %d\n", metrics.StaleContexts)

// Storage metrics
fmt.Printf("Stored contexts: %d\n", metrics.StoredContexts)
fmt.Printf("Encrypted contexts: %d\n", metrics.EncryptedContexts)
fmt.Printf("Storage utilization: %.2f%%\n", metrics.StorageUtilization*100)

// Intelligence metrics
fmt.Printf("Generation requests: %d\n", metrics.GenerationRequests)
fmt.Printf("Successful generations: %d\n", metrics.SuccessfulGenerations)
fmt.Printf("Pattern matches: %d\n", metrics.PatternMatches)

Event Monitoring

// Monitor system events
slurp.RegisterEventHandler(EventContextResolved, metricsCollector)
slurp.RegisterEventHandler(EventContextGenerated, auditLogger)
slurp.RegisterEventHandler(EventErrorOccurred, errorTracker)
slurp.RegisterEventHandler(EventStalenessDetected, alertSystem)

Implementation Status

Completed Features

• Core SLURP Coordinator: Production-ready main coordinator
• Context Resolution: Bounded hierarchy traversal with caching
• Temporal Graph: Decision-hop temporal analysis fully implemented
• Storage Layer: Local and distributed storage operational
• Leader Integration: Election-based leader coordination working
• Role-Based Security: Encryption and access control functional
• Event System: Event handling and notification working
• Metrics Collection: Performance monitoring active

In Development

• Alignment Analyzer: Goal alignment assessment (stubs in place)
• Intelligence Engine: Context generation engine (partial implementation)
• Distribution Layer: Full DHT-based distribution (partial)
• Pattern Detection: Advanced pattern matching capabilities
• Query Optimization: Advanced query and search features

Experimental Features

• RAG Integration: External RAG system integration (experimental)
• Multi-language Analysis: Beyond Go language support
• Graph Visualization: Temporal graph visualization tools
• ML-Based Staleness: Machine learning for staleness prediction
• Automated Repair: Self-healing context inconsistencies

Troubleshooting

Common Issues

Issue: Context Not Found

// Symptom
resolved, err := slurp.Resolve(ctx, address)
// Returns: "context not found for ucxl://..."

// Causes:
// 1. Context never generated for this address
// 2. Cache invalidated and persistence not enabled
// 3. Role permissions prevent access

// Solutions:
// 1. Generate context (if admin)
if slurp.IsCurrentNodeAdmin() {
    generated, err := slurp.GenerateContext(ctx, path, options)
}

// 2. Check role permissions
// 3. Verify storage configuration

Issue: High Resolution Latency

// Symptom: Slow context resolution (> 1 second)

// Causes:
// 1. Cache disabled or not warming up
// 2. Deep hierarchy traversal
// 3. Network issues with DHT
// 4. Storage backend slow

// Solutions:
// 1. Enable caching with appropriate TTL
config.Slurp.ContextResolution.CacheTTL = 15 * time.Minute

// 2. Reduce depth limit
resolved, err := slurp.ResolveWithDepth(ctx, address, 3)

// 3. Use batch resolution
results, err := slurp.BatchResolve(ctx, addresses)

// 4. Check storage metrics
metrics := slurp.GetMetrics()
fmt.Printf("Cache hit rate: %.2f%%\n", metrics.CacheHitRate*100)

Issue: Admin Node Not Generating Context

// Symptom: Context generation fails with "requires admin privileges"

// Causes:
// 1. Node not elected as admin
// 2. Election system not initialized
// 3. Leadership change in progress

// Solutions:
// 1. Check admin status
if !slurp.IsCurrentNodeAdmin() {
    fmt.Printf("Current admin: %s\n", slurp.currentAdmin)
    // Wait for election or request from admin
}

// 2. Verify election system
if election.GetCurrentAdmin() == "" {
    // No admin elected yet
}

// 3. Monitor admin changes
slurp.RegisterEventHandler(EventAdminChanged, handler)

Issue: Temporal Navigation Returns No Results

// Symptom: GetTemporalEvolution returns empty array

// Causes:
// 1. Temporal tracking not enabled
// 2. No evolution recorded for this context
// 3. Temporal storage not initialized

// Solutions:
// 1. Evolve context when changes occur
decision := &DecisionMetadata{/*...*/}
evolved, err := slurp.temporalGraph.EvolveContext(ctx, address, newContext, reason, decision)

// 2. Check temporal system initialization
if slurp.temporalGraph == nil {
    // Temporal system not initialized
}

// 3. Verify temporal storage
if slurp.temporalStore == nil {
    // Storage not configured
}

Related Packages

• pkg/dht: Distributed Hash Table for storage
• pkg/election: Leader election for coordination
• pkg/crypto: Role-based encryption and access control
• pkg/ucxl: UCXL address parsing and handling
• pkg/config: Configuration management

Subpackage Documentation

Detailed documentation for each subpackage:

• alignment/ - Goal alignment assessment and tracking
• context/ - Hierarchical context resolution
• distribution/ - Distributed context sharing
• intelligence/ - AI-powered context generation
• leader/ - Leader-based coordination
• roles/ - Role-based access control
• storage/ - Persistence and caching layer
• temporal/ - Decision-hop temporal analysis

Further Reading

• CHORUS Architecture Documentation
• DHT Design and Implementation
• Election System Documentation
• Role-Based Access Control Guide
• UCXL Address Specification