bzzz/archive/SLURP_GO_ARCHITECTURE_DESIGN.md

# SLURP Contextual Intelligence System - Go Architecture Design

## Overview

This document provides the complete architectural design for implementing the SLURP (Storage, Logic, Understanding, Retrieval, Processing) contextual intelligence system in Go, integrated with the existing BZZZ infrastructure.

## Current BZZZ Architecture Analysis

### Existing Package Structure
```
pkg/
├── config/          # Configuration management
├── crypto/          # Encryption, Shamir's Secret Sharing
├── dht/            # Distributed Hash Table (mock + real)
├── election/       # Leader election algorithms
├── types/          # Common types and interfaces
├── ucxl/           # UCXL address parsing and handling
└── ...
```

### Key Integration Points
- **DHT Integration**: `pkg/dht/` for context distribution
- **Crypto Integration**: `pkg/crypto/` for role-based encryption
- **Election Integration**: `pkg/election/` for Leader duties
- **UCXL Integration**: `pkg/ucxl/` for address parsing
- **Config Integration**: `pkg/config/` for system configuration

## Go Package Design

### Package Structure
```
pkg/slurp/
├── context/         # Core context types and interfaces
├── intelligence/    # Context analysis and generation
├── storage/         # Context persistence and retrieval
├── distribution/    # Context network distribution
├── temporal/        # Decision-hop temporal analysis
├── alignment/       # Project goal alignment
├── roles/          # Role-based access control
└── leader/         # Leader-specific context duties
```

## Core Types and Interfaces

### 1. Context Types (`pkg/slurp/context/types.go`)

```go
package context

import (
    "time"
    "github.com/your-org/bzzz/pkg/ucxl"
    "github.com/your-org/bzzz/pkg/types"
)

// ContextNode represents a hierarchical context node
type ContextNode struct {
    Path                string                 `json:"path"`
    UCXLAddress         ucxl.Address          `json:"ucxl_address"`
    Summary             string                 `json:"summary"`
    Purpose             string                 `json:"purpose"`
    Technologies        []string              `json:"technologies"`
    Tags                []string              `json:"tags"`
    Insights            []string              `json:"insights"`
    
    // Hierarchy control
    OverridesParent     bool                  `json:"overrides_parent"`
    ContextSpecificity  int                   `json:"context_specificity"`
    AppliesToChildren   bool                  `json:"applies_to_children"`
    
    // Metadata
    GeneratedAt         time.Time             `json:"generated_at"`
    RAGConfidence       float64               `json:"rag_confidence"`
}

// RoleAccessLevel defines encryption levels for different roles
type RoleAccessLevel int

const (
    AccessPublic RoleAccessLevel = iota
    AccessLow
    AccessMedium
    AccessHigh
    AccessCritical
)

// EncryptedContext represents role-encrypted context data
type EncryptedContext struct {
    UCXLAddress     ucxl.Address      `json:"ucxl_address"`
    Role            string            `json:"role"`
    AccessLevel     RoleAccessLevel   `json:"access_level"`
    EncryptedData   []byte            `json:"encrypted_data"`
    KeyFingerprint  string            `json:"key_fingerprint"`
    CreatedAt       time.Time         `json:"created_at"`
}

// ResolvedContext is the final resolved context for consumption
type ResolvedContext struct {
    UCXLAddress         ucxl.Address  `json:"ucxl_address"`
    Summary             string        `json:"summary"`
    Purpose             string        `json:"purpose"`
    Technologies        []string      `json:"technologies"`
    Tags                []string      `json:"tags"`
    Insights            []string      `json:"insights"`
    
    // Resolution metadata
    ContextSourcePath   string        `json:"context_source_path"`
    InheritanceChain    []string      `json:"inheritance_chain"`
    ResolutionConfidence float64      `json:"resolution_confidence"`
    BoundedDepth        int           `json:"bounded_depth"`
    GlobalContextsApplied bool        `json:"global_contexts_applied"`
    ResolvedAt          time.Time     `json:"resolved_at"`
}
```

### 2. Context Resolver Interface (`pkg/slurp/context/resolver.go`)

```go
package context

// ContextResolver defines the interface for hierarchical context resolution
type ContextResolver interface {
    // Resolve context for a UCXL address with bounded hierarchy traversal
    Resolve(address ucxl.Address, role string, maxDepth int) (*ResolvedContext, error)
    
    // Add global context that applies to all addresses
    AddGlobalContext(ctx *ContextNode) error
    
    // Set maximum hierarchy depth for bounded traversal
    SetHierarchyDepthLimit(maxDepth int)
    
    // Get resolution statistics
    GetStatistics() *ResolutionStatistics
}

type ResolutionStatistics struct {
    ContextNodes        int           `json:"context_nodes"`
    GlobalContexts      int           `json:"global_contexts"`
    MaxHierarchyDepth   int           `json:"max_hierarchy_depth"`
    CachedResolutions   int           `json:"cached_resolutions"`
    TotalResolutions    int           `json:"total_resolutions"`
}
```

### 3. Temporal Decision Analysis (`pkg/slurp/temporal/types.go`)

```go
package temporal

import (
    "time"
    "github.com/your-org/bzzz/pkg/ucxl"
)

// ChangeReason represents why a context changed
type ChangeReason string

const (
    InitialCreation     ChangeReason = "initial_creation"
    CodeChange          ChangeReason = "code_change"
    DesignDecision      ChangeReason = "design_decision"
    Refactoring         ChangeReason = "refactoring"
    ArchitectureChange  ChangeReason = "architecture_change"
    RequirementsChange  ChangeReason = "requirements_change"
    LearningEvolution   ChangeReason = "learning_evolution"
    RAGEnhancement      ChangeReason = "rag_enhancement"
    TeamInput           ChangeReason = "team_input"
)

// DecisionMetadata captures information about a decision
type DecisionMetadata struct {
    DecisionMaker       string    `json:"decision_maker"`
    DecisionID          string    `json:"decision_id"` // Git commit, ticket ID, etc.
    DecisionRationale   string    `json:"decision_rationale"`
    ImpactScope         string    `json:"impact_scope"` // local, module, project, system
    ConfidenceLevel     float64   `json:"confidence_level"`
    ExternalReferences  []string  `json:"external_references"`
    Timestamp           time.Time `json:"timestamp"`
}

// TemporalContextNode represents context at a specific decision point
type TemporalContextNode struct {
    UCXLAddress     ucxl.Address      `json:"ucxl_address"`
    Version         int               `json:"version"`
    
    // Core context (embedded)
    Context         *ContextNode      `json:"context"`
    
    // Temporal metadata
    ChangeReason    ChangeReason      `json:"change_reason"`
    ParentVersion   *int              `json:"parent_version,omitempty"`
    DecisionMeta    *DecisionMetadata `json:"decision_metadata"`
    
    // Evolution tracking
    ContextHash     string            `json:"context_hash"`
    ConfidenceScore float64           `json:"confidence_score"`
    StalenessScore  float64           `json:"staleness_score"`
    
    // Decision influence graph
    Influences      []ucxl.Address    `json:"influences"`       // Addresses this decision affects
    InfluencedBy    []ucxl.Address    `json:"influenced_by"`    // Addresses that affect this
}

// DecisionPath represents a path between two decisions
type DecisionPath struct {
    FromAddress ucxl.Address             `json:"from_address"`
    ToAddress   ucxl.Address             `json:"to_address"`
    Path        []*TemporalContextNode   `json:"path"`
    HopDistance int                      `json:"hop_distance"`
}
```

### 4. Intelligence Engine Interface (`pkg/slurp/intelligence/engine.go`)

```go
package intelligence

import (
    "context"
    "github.com/your-org/bzzz/pkg/ucxl"
    slurpContext "github.com/your-org/bzzz/pkg/slurp/context"
)

// IntelligenceEngine generates contextual understanding
type IntelligenceEngine interface {
    // Analyze a filesystem path and generate context
    AnalyzeFile(ctx context.Context, filePath string, role string) (*slurpContext.ContextNode, error)
    
    // Analyze directory structure for hierarchical patterns
    AnalyzeDirectory(ctx context.Context, dirPath string) ([]*slurpContext.ContextNode, error)
    
    // Generate role-specific insights
    GenerateRoleInsights(ctx context.Context, baseContext *slurpContext.ContextNode, role string) ([]string, error)
    
    // Assess project goal alignment
    AssessGoalAlignment(ctx context.Context, node *slurpContext.ContextNode) (float64, error)
}

// ProjectGoal represents a high-level project objective
type ProjectGoal struct {
    ID          string   `json:"id"`
    Name        string   `json:"name"`
    Description string   `json:"description"`
    Keywords    []string `json:"keywords"`
    Priority    int      `json:"priority"`
    Phase       string   `json:"phase"`
}

// RoleProfile defines what context a role needs
type RoleProfile struct {
    Role            string                      `json:"role"`
    AccessLevel     slurpContext.RoleAccessLevel `json:"access_level"`
    RelevantTags    []string                    `json:"relevant_tags"`
    ContextScope    []string                    `json:"context_scope"` // frontend, backend, infrastructure, etc.
    InsightTypes    []string                    `json:"insight_types"`
}
```

### 5. Leader Integration (`pkg/slurp/leader/manager.go`)

```go
package leader

import (
    "context"
    "sync"
    
    "github.com/your-org/bzzz/pkg/election"
    "github.com/your-org/bzzz/pkg/dht"
    "github.com/your-org/bzzz/pkg/slurp/intelligence"
    slurpContext "github.com/your-org/bzzz/pkg/slurp/context"
)

// ContextManager handles leader-only context generation duties
type ContextManager struct {
    mu              sync.RWMutex
    isLeader        bool
    election        election.Election
    dht             dht.DHT
    intelligence    intelligence.IntelligenceEngine
    contextResolver slurpContext.ContextResolver
    
    // Context generation state
    generationQueue chan *ContextGenerationRequest
    activeJobs      map[string]*ContextGenerationJob
}

type ContextGenerationRequest struct {
    UCXLAddress ucxl.Address    `json:"ucxl_address"`
    FilePath    string          `json:"file_path"`
    Priority    int             `json:"priority"`
    RequestedBy string          `json:"requested_by"`
    Role        string          `json:"role"`
}

type ContextGenerationJob struct {
    Request     *ContextGenerationRequest
    Status      JobStatus
    StartedAt   time.Time
    CompletedAt *time.Time
    Result      *slurpContext.ContextNode
    Error       error
}

type JobStatus string

const (
    JobPending    JobStatus = "pending"
    JobRunning    JobStatus = "running"
    JobCompleted  JobStatus = "completed"
    JobFailed     JobStatus = "failed"
)

// NewContextManager creates a new leader context manager
func NewContextManager(
    election election.Election,
    dht dht.DHT,
    intelligence intelligence.IntelligenceEngine,
    resolver slurpContext.ContextResolver,
) *ContextManager {
    cm := &ContextManager{
        election:        election,
        dht:            dht,
        intelligence:   intelligence,
        contextResolver: resolver,
        generationQueue: make(chan *ContextGenerationRequest, 1000),
        activeJobs:     make(map[string]*ContextGenerationJob),
    }
    
    // Listen for leadership changes
    go cm.watchLeadershipChanges()
    
    // Process context generation requests (only when leader)
    go cm.processContextGeneration()
    
    return cm
}

// RequestContextGeneration queues a context generation request
func (cm *ContextManager) RequestContextGeneration(req *ContextGenerationRequest) error {
    select {
    case cm.generationQueue <- req:
        return nil
    default:
        return errors.New("context generation queue is full")
    }
}

// IsLeader returns whether this node is the current leader
func (cm *ContextManager) IsLeader() bool {
    cm.mu.RLock()
    defer cm.mu.RUnlock()
    return cm.isLeader
}
```

## Integration with Existing BZZZ Systems

### 1. DHT Integration (`pkg/slurp/distribution/dht.go`)

```go
package distribution

import (
    "github.com/your-org/bzzz/pkg/dht"
    "github.com/your-org/bzzz/pkg/crypto"
    slurpContext "github.com/your-org/bzzz/pkg/slurp/context"
)

// ContextDistributor handles context distribution through DHT
type ContextDistributor struct {
    dht     dht.DHT
    crypto  crypto.Crypto
}

// DistributeContext encrypts and stores context in DHT for role-based access
func (cd *ContextDistributor) DistributeContext(
    ctx *slurpContext.ContextNode, 
    roles []string,
) error {
    // For each role, encrypt the context with role-specific keys
    for _, role := range roles {
        encryptedCtx, err := cd.encryptForRole(ctx, role)
        if err != nil {
            return fmt.Errorf("failed to encrypt context for role %s: %w", role, err)
        }
        
        // Store in DHT with role-specific key
        key := cd.generateContextKey(ctx.UCXLAddress, role)
        if err := cd.dht.Put(key, encryptedCtx); err != nil {
            return fmt.Errorf("failed to store context in DHT: %w", err)
        }
    }
    
    return nil
}

// RetrieveContext gets context from DHT and decrypts for the requesting role
func (cd *ContextDistributor) RetrieveContext(
    address ucxl.Address, 
    role string,
) (*slurpContext.ResolvedContext, error) {
    key := cd.generateContextKey(address, role)
    
    encryptedData, err := cd.dht.Get(key)
    if err != nil {
        return nil, fmt.Errorf("failed to retrieve context from DHT: %w", err)
    }
    
    return cd.decryptForRole(encryptedData, role)
}
```

### 2. Configuration Integration (`pkg/slurp/config/config.go`)

```go
package config

import "github.com/your-org/bzzz/pkg/config"

// SLURPConfig extends BZZZ config with SLURP-specific settings
type SLURPConfig struct {
    // Context generation settings
    MaxHierarchyDepth       int     `yaml:"max_hierarchy_depth" json:"max_hierarchy_depth"`
    ContextCacheTTL         int     `yaml:"context_cache_ttl" json:"context_cache_ttl"`
    GenerationConcurrency   int     `yaml:"generation_concurrency" json:"generation_concurrency"`
    
    // Role-based access
    RoleProfiles           map[string]*RoleProfile `yaml:"role_profiles" json:"role_profiles"`
    DefaultAccessLevel     string                  `yaml:"default_access_level" json:"default_access_level"`
    
    // Intelligence engine settings
    RAGEndpoint            string  `yaml:"rag_endpoint" json:"rag_endpoint"`
    RAGTimeout             int     `yaml:"rag_timeout" json:"rag_timeout"`
    ConfidenceThreshold    float64 `yaml:"confidence_threshold" json:"confidence_threshold"`
    
    // Project goals
    ProjectGoals           []*ProjectGoal `yaml:"project_goals" json:"project_goals"`
}

// LoadSLURPConfig extends the main BZZZ config loading
func LoadSLURPConfig(configPath string) (*config.Config, error) {
    // Load base BZZZ config
    bzzzConfig, err := config.Load(configPath)
    if err != nil {
        return nil, err
    }
    
    // Load SLURP-specific extensions
    slurpConfig := &SLURPConfig{}
    if err := config.LoadSection("slurp", slurpConfig); err != nil {
        // Use defaults if SLURP config not found
        slurpConfig = DefaultSLURPConfig()
    }
    
    // Merge into main config
    bzzzConfig.SLURP = slurpConfig
    return bzzzConfig, nil
}
```

## Implementation Phases

### Phase 1: Foundation (Week 1-2)
1. **Create base package structure** in `pkg/slurp/`
2. **Define core interfaces and types** (`context`, `temporal`)
3. **Integrate with existing election system** for leader duties
4. **Basic context resolver implementation** with bounded traversal

### Phase 2: Encryption & Distribution (Week 3-4)  
1. **Extend crypto package** for role-based encryption
2. **Implement DHT context distribution** 
3. **Role-based access control** integration
4. **Context caching and retrieval**

### Phase 3: Intelligence Engine (Week 5-7)
1. **File analysis and context generation**
2. **Decision temporal graph implementation**
3. **Project goal alignment** 
4. **RAG integration** for enhanced context

### Phase 4: Integration & Testing (Week 8)
1. **End-to-end integration testing**
2. **Performance optimization**
3. **Documentation and examples**
4. **Leader failover testing**

## Key Go Patterns Used

### 1. Interface-Driven Design
All major components define clear interfaces, allowing for testing and future extensibility.

### 2. Context Propagation
Using Go's `context` package for cancellation and timeouts throughout the system.

### 3. Concurrent Processing
Goroutines and channels for context generation queue processing and distributed operations.

### 4. Error Handling
Proper error wrapping and handling following Go best practices.

### 5. Configuration
Extends existing BZZZ configuration patterns seamlessly.

## Migration from Python Prototypes

The Python prototypes provide the algorithmic foundation:

1. **Bounded hierarchy walking** → Go recursive traversal with depth limits
2. **CSS-like context inheritance** → Go struct composition and merging
3. **Decision-hop analysis** → Go graph algorithms and BFS traversal
4. **Role-based encryption** → Integration with existing Go crypto package
5. **Temporal versioning** → Go time handling and version management

## Next Steps After Restart

1. **Run the systems-engineer agent** to create the Go package structure
2. **Implement core interfaces** starting with `pkg/slurp/context/`
3. **Integrate with existing BZZZ systems** step by step
4. **Test each component** as it's implemented
5. **Build up to full Leader-coordinated context generation**

This design ensures the SLURP system feels like a native part of BZZZ while providing the sophisticated contextual intelligence capabilities we designed.