bzzz/archive/SLURP_GO_ARCHITECTURE_DESIGN.md

SLURP Contextual Intelligence System - Go Architecture Design

Overview

This document provides the complete architectural design for implementing the SLURP (Storage, Logic, Understanding, Retrieval, Processing) contextual intelligence system in Go, integrated with the existing BZZZ infrastructure.

Current BZZZ Architecture Analysis

Existing Package Structure

pkg/
├── config/          # Configuration management
├── crypto/          # Encryption, Shamir's Secret Sharing
├── dht/            # Distributed Hash Table (mock + real)
├── election/       # Leader election algorithms
├── types/          # Common types and interfaces
├── ucxl/           # UCXL address parsing and handling
└── ...

Key Integration Points

• DHT Integration: pkg/dht/ for context distribution
• Crypto Integration: pkg/crypto/ for role-based encryption
• Election Integration: pkg/election/ for Leader duties
• UCXL Integration: pkg/ucxl/ for address parsing
• Config Integration: pkg/config/ for system configuration

Go Package Design

Package Structure

pkg/slurp/
├── context/         # Core context types and interfaces
├── intelligence/    # Context analysis and generation
├── storage/         # Context persistence and retrieval
├── distribution/    # Context network distribution
├── temporal/        # Decision-hop temporal analysis
├── alignment/       # Project goal alignment
├── roles/          # Role-based access control
└── leader/         # Leader-specific context duties

Core Types and Interfaces

1. Context Types (pkg/slurp/context/types.go)

package context

import (
    "time"
    "github.com/your-org/bzzz/pkg/ucxl"
    "github.com/your-org/bzzz/pkg/types"
)

// ContextNode represents a hierarchical context node
type ContextNode struct {
    Path                string                 `json:"path"`
    UCXLAddress         ucxl.Address          `json:"ucxl_address"`
    Summary             string                 `json:"summary"`
    Purpose             string                 `json:"purpose"`
    Technologies        []string              `json:"technologies"`
    Tags                []string              `json:"tags"`
    Insights            []string              `json:"insights"`
    
    // Hierarchy control
    OverridesParent     bool                  `json:"overrides_parent"`
    ContextSpecificity  int                   `json:"context_specificity"`
    AppliesToChildren   bool                  `json:"applies_to_children"`
    
    // Metadata
    GeneratedAt         time.Time             `json:"generated_at"`
    RAGConfidence       float64               `json:"rag_confidence"`
}

// RoleAccessLevel defines encryption levels for different roles
type RoleAccessLevel int

const (
    AccessPublic RoleAccessLevel = iota
    AccessLow
    AccessMedium
    AccessHigh
    AccessCritical
)

// EncryptedContext represents role-encrypted context data
type EncryptedContext struct {
    UCXLAddress     ucxl.Address      `json:"ucxl_address"`
    Role            string            `json:"role"`
    AccessLevel     RoleAccessLevel   `json:"access_level"`
    EncryptedData   []byte            `json:"encrypted_data"`
    KeyFingerprint  string            `json:"key_fingerprint"`
    CreatedAt       time.Time         `json:"created_at"`
}

// ResolvedContext is the final resolved context for consumption
type ResolvedContext struct {
    UCXLAddress         ucxl.Address  `json:"ucxl_address"`
    Summary             string        `json:"summary"`
    Purpose             string        `json:"purpose"`
    Technologies        []string      `json:"technologies"`
    Tags                []string      `json:"tags"`
    Insights            []string      `json:"insights"`
    
    // Resolution metadata
    ContextSourcePath   string        `json:"context_source_path"`
    InheritanceChain    []string      `json:"inheritance_chain"`
    ResolutionConfidence float64      `json:"resolution_confidence"`
    BoundedDepth        int           `json:"bounded_depth"`
    GlobalContextsApplied bool        `json:"global_contexts_applied"`
    ResolvedAt          time.Time     `json:"resolved_at"`
}

2. Context Resolver Interface (pkg/slurp/context/resolver.go)

package context

// ContextResolver defines the interface for hierarchical context resolution
type ContextResolver interface {
    // Resolve context for a UCXL address with bounded hierarchy traversal
    Resolve(address ucxl.Address, role string, maxDepth int) (*ResolvedContext, error)
    
    // Add global context that applies to all addresses
    AddGlobalContext(ctx *ContextNode) error
    
    // Set maximum hierarchy depth for bounded traversal
    SetHierarchyDepthLimit(maxDepth int)
    
    // Get resolution statistics
    GetStatistics() *ResolutionStatistics
}

type ResolutionStatistics struct {
    ContextNodes        int           `json:"context_nodes"`
    GlobalContexts      int           `json:"global_contexts"`
    MaxHierarchyDepth   int           `json:"max_hierarchy_depth"`
    CachedResolutions   int           `json:"cached_resolutions"`
    TotalResolutions    int           `json:"total_resolutions"`
}

3. Temporal Decision Analysis (pkg/slurp/temporal/types.go)

package temporal

import (
    "time"
    "github.com/your-org/bzzz/pkg/ucxl"
)

// ChangeReason represents why a context changed
type ChangeReason string

const (
    InitialCreation     ChangeReason = "initial_creation"
    CodeChange          ChangeReason = "code_change"
    DesignDecision      ChangeReason = "design_decision"
    Refactoring         ChangeReason = "refactoring"
    ArchitectureChange  ChangeReason = "architecture_change"
    RequirementsChange  ChangeReason = "requirements_change"
    LearningEvolution   ChangeReason = "learning_evolution"
    RAGEnhancement      ChangeReason = "rag_enhancement"
    TeamInput           ChangeReason = "team_input"
)

// DecisionMetadata captures information about a decision
type DecisionMetadata struct {
    DecisionMaker       string    `json:"decision_maker"`
    DecisionID          string    `json:"decision_id"` // Git commit, ticket ID, etc.
    DecisionRationale   string    `json:"decision_rationale"`
    ImpactScope         string    `json:"impact_scope"` // local, module, project, system
    ConfidenceLevel     float64   `json:"confidence_level"`
    ExternalReferences  []string  `json:"external_references"`
    Timestamp           time.Time `json:"timestamp"`
}

// TemporalContextNode represents context at a specific decision point
type TemporalContextNode struct {
    UCXLAddress     ucxl.Address      `json:"ucxl_address"`
    Version         int               `json:"version"`
    
    // Core context (embedded)
    Context         *ContextNode      `json:"context"`
    
    // Temporal metadata
    ChangeReason    ChangeReason      `json:"change_reason"`
    ParentVersion   *int              `json:"parent_version,omitempty"`
    DecisionMeta    *DecisionMetadata `json:"decision_metadata"`
    
    // Evolution tracking
    ContextHash     string            `json:"context_hash"`
    ConfidenceScore float64           `json:"confidence_score"`
    StalenessScore  float64           `json:"staleness_score"`
    
    // Decision influence graph
    Influences      []ucxl.Address    `json:"influences"`       // Addresses this decision affects
    InfluencedBy    []ucxl.Address    `json:"influenced_by"`    // Addresses that affect this
}

// DecisionPath represents a path between two decisions
type DecisionPath struct {
    FromAddress ucxl.Address             `json:"from_address"`
    ToAddress   ucxl.Address             `json:"to_address"`
    Path        []*TemporalContextNode   `json:"path"`
    HopDistance int                      `json:"hop_distance"`
}

4. Intelligence Engine Interface (pkg/slurp/intelligence/engine.go)

package intelligence

import (
    "context"
    "github.com/your-org/bzzz/pkg/ucxl"
    slurpContext "github.com/your-org/bzzz/pkg/slurp/context"
)

// IntelligenceEngine generates contextual understanding
type IntelligenceEngine interface {
    // Analyze a filesystem path and generate context
    AnalyzeFile(ctx context.Context, filePath string, role string) (*slurpContext.ContextNode, error)
    
    // Analyze directory structure for hierarchical patterns
    AnalyzeDirectory(ctx context.Context, dirPath string) ([]*slurpContext.ContextNode, error)
    
    // Generate role-specific insights
    GenerateRoleInsights(ctx context.Context, baseContext *slurpContext.ContextNode, role string) ([]string, error)
    
    // Assess project goal alignment
    AssessGoalAlignment(ctx context.Context, node *slurpContext.ContextNode) (float64, error)
}

// ProjectGoal represents a high-level project objective
type ProjectGoal struct {
    ID          string   `json:"id"`
    Name        string   `json:"name"`
    Description string   `json:"description"`
    Keywords    []string `json:"keywords"`
    Priority    int      `json:"priority"`
    Phase       string   `json:"phase"`
}

// RoleProfile defines what context a role needs
type RoleProfile struct {
    Role            string                      `json:"role"`
    AccessLevel     slurpContext.RoleAccessLevel `json:"access_level"`
    RelevantTags    []string                    `json:"relevant_tags"`
    ContextScope    []string                    `json:"context_scope"` // frontend, backend, infrastructure, etc.
    InsightTypes    []string                    `json:"insight_types"`
}

5. Leader Integration (pkg/slurp/leader/manager.go)

package leader

import (
    "context"
    "sync"
    
    "github.com/your-org/bzzz/pkg/election"
    "github.com/your-org/bzzz/pkg/dht"
    "github.com/your-org/bzzz/pkg/slurp/intelligence"
    slurpContext "github.com/your-org/bzzz/pkg/slurp/context"
)

// ContextManager handles leader-only context generation duties
type ContextManager struct {
    mu              sync.RWMutex
    isLeader        bool
    election        election.Election
    dht             dht.DHT
    intelligence    intelligence.IntelligenceEngine
    contextResolver slurpContext.ContextResolver
    
    // Context generation state
    generationQueue chan *ContextGenerationRequest
    activeJobs      map[string]*ContextGenerationJob
}

type ContextGenerationRequest struct {
    UCXLAddress ucxl.Address    `json:"ucxl_address"`
    FilePath    string          `json:"file_path"`
    Priority    int             `json:"priority"`
    RequestedBy string          `json:"requested_by"`
    Role        string          `json:"role"`
}

type ContextGenerationJob struct {
    Request     *ContextGenerationRequest
    Status      JobStatus
    StartedAt   time.Time
    CompletedAt *time.Time
    Result      *slurpContext.ContextNode
    Error       error
}

type JobStatus string

const (
    JobPending    JobStatus = "pending"
    JobRunning    JobStatus = "running"
    JobCompleted  JobStatus = "completed"
    JobFailed     JobStatus = "failed"
)

// NewContextManager creates a new leader context manager
func NewContextManager(
    election election.Election,
    dht dht.DHT,
    intelligence intelligence.IntelligenceEngine,
    resolver slurpContext.ContextResolver,
) *ContextManager {
    cm := &ContextManager{
        election:        election,
        dht:            dht,
        intelligence:   intelligence,
        contextResolver: resolver,
        generationQueue: make(chan *ContextGenerationRequest, 1000),
        activeJobs:     make(map[string]*ContextGenerationJob),
    }
    
    // Listen for leadership changes
    go cm.watchLeadershipChanges()
    
    // Process context generation requests (only when leader)
    go cm.processContextGeneration()
    
    return cm
}

// RequestContextGeneration queues a context generation request
func (cm *ContextManager) RequestContextGeneration(req *ContextGenerationRequest) error {
    select {
    case cm.generationQueue <- req:
        return nil
    default:
        return errors.New("context generation queue is full")
    }
}

// IsLeader returns whether this node is the current leader
func (cm *ContextManager) IsLeader() bool {
    cm.mu.RLock()
    defer cm.mu.RUnlock()
    return cm.isLeader
}

Integration with Existing BZZZ Systems

1. DHT Integration (pkg/slurp/distribution/dht.go)

package distribution

import (
    "github.com/your-org/bzzz/pkg/dht"
    "github.com/your-org/bzzz/pkg/crypto"
    slurpContext "github.com/your-org/bzzz/pkg/slurp/context"
)

// ContextDistributor handles context distribution through DHT
type ContextDistributor struct {
    dht     dht.DHT
    crypto  crypto.Crypto
}

// DistributeContext encrypts and stores context in DHT for role-based access
func (cd *ContextDistributor) DistributeContext(
    ctx *slurpContext.ContextNode, 
    roles []string,
) error {
    // For each role, encrypt the context with role-specific keys
    for _, role := range roles {
        encryptedCtx, err := cd.encryptForRole(ctx, role)
        if err != nil {
            return fmt.Errorf("failed to encrypt context for role %s: %w", role, err)
        }
        
        // Store in DHT with role-specific key
        key := cd.generateContextKey(ctx.UCXLAddress, role)
        if err := cd.dht.Put(key, encryptedCtx); err != nil {
            return fmt.Errorf("failed to store context in DHT: %w", err)
        }
    }
    
    return nil
}

// RetrieveContext gets context from DHT and decrypts for the requesting role
func (cd *ContextDistributor) RetrieveContext(
    address ucxl.Address, 
    role string,
) (*slurpContext.ResolvedContext, error) {
    key := cd.generateContextKey(address, role)
    
    encryptedData, err := cd.dht.Get(key)
    if err != nil {
        return nil, fmt.Errorf("failed to retrieve context from DHT: %w", err)
    }
    
    return cd.decryptForRole(encryptedData, role)
}

2. Configuration Integration (pkg/slurp/config/config.go)

package config

import "github.com/your-org/bzzz/pkg/config"

// SLURPConfig extends BZZZ config with SLURP-specific settings
type SLURPConfig struct {
    // Context generation settings
    MaxHierarchyDepth       int     `yaml:"max_hierarchy_depth" json:"max_hierarchy_depth"`
    ContextCacheTTL         int     `yaml:"context_cache_ttl" json:"context_cache_ttl"`
    GenerationConcurrency   int     `yaml:"generation_concurrency" json:"generation_concurrency"`
    
    // Role-based access
    RoleProfiles           map[string]*RoleProfile `yaml:"role_profiles" json:"role_profiles"`
    DefaultAccessLevel     string                  `yaml:"default_access_level" json:"default_access_level"`
    
    // Intelligence engine settings
    RAGEndpoint            string  `yaml:"rag_endpoint" json:"rag_endpoint"`
    RAGTimeout             int     `yaml:"rag_timeout" json:"rag_timeout"`
    ConfidenceThreshold    float64 `yaml:"confidence_threshold" json:"confidence_threshold"`
    
    // Project goals
    ProjectGoals           []*ProjectGoal `yaml:"project_goals" json:"project_goals"`
}

// LoadSLURPConfig extends the main BZZZ config loading
func LoadSLURPConfig(configPath string) (*config.Config, error) {
    // Load base BZZZ config
    bzzzConfig, err := config.Load(configPath)
    if err != nil {
        return nil, err
    }
    
    // Load SLURP-specific extensions
    slurpConfig := &SLURPConfig{}
    if err := config.LoadSection("slurp", slurpConfig); err != nil {
        // Use defaults if SLURP config not found
        slurpConfig = DefaultSLURPConfig()
    }
    
    // Merge into main config
    bzzzConfig.SLURP = slurpConfig
    return bzzzConfig, nil
}

Implementation Phases

Phase 1: Foundation (Week 1-2)

1. Create base package structure in pkg/slurp/
2. Define core interfaces and types (context, temporal)
3. Integrate with existing election system for leader duties
4. Basic context resolver implementation with bounded traversal

Phase 2: Encryption & Distribution (Week 3-4)

1. Extend crypto package for role-based encryption
2. Implement DHT context distribution
3. Role-based access control integration
4. Context caching and retrieval

Phase 3: Intelligence Engine (Week 5-7)

1. File analysis and context generation
2. Decision temporal graph implementation
3. Project goal alignment
4. RAG integration for enhanced context

Phase 4: Integration & Testing (Week 8)

1. End-to-end integration testing
2. Performance optimization
3. Documentation and examples
4. Leader failover testing

Key Go Patterns Used

1. Interface-Driven Design

All major components define clear interfaces, allowing for testing and future extensibility.

2. Context Propagation

Using Go's context package for cancellation and timeouts throughout the system.

3. Concurrent Processing

Goroutines and channels for context generation queue processing and distributed operations.

4. Error Handling

Proper error wrapping and handling following Go best practices.

5. Configuration

Extends existing BZZZ configuration patterns seamlessly.

Migration from Python Prototypes

The Python prototypes provide the algorithmic foundation:

1. Bounded hierarchy walking → Go recursive traversal with depth limits
2. CSS-like context inheritance → Go struct composition and merging
3. Decision-hop analysis → Go graph algorithms and BFS traversal
4. Role-based encryption → Integration with existing Go crypto package
5. Temporal versioning → Go time handling and version management

Next Steps After Restart

1. Run the systems-engineer agent to create the Go package structure
2. Implement core interfaces starting with pkg/slurp/context/
3. Integrate with existing BZZZ systems step by step
4. Test each component as it's implemented
5. Build up to full Leader-coordinated context generation

This design ensures the SLURP system feels like a native part of BZZZ while providing the sophisticated contextual intelligence capabilities we designed.