# SLURP Go Architecture Specification ## Executive Summary This document specifies the Go-based SLURP (Storage, Logic, Understanding, Retrieval, Processing) system architecture for BZZZ, translating the Python prototypes into native Go packages that integrate seamlessly with the existing BZZZ distributed system. **SLURP implements contextual intelligence capabilities:** - **Storage**: Hierarchical context metadata storage with bounded depth traversal - **Logic**: Decision-hop temporal analysis for tracking conceptual evolution - **Understanding**: Cascading context resolution with role-based encryption - **Retrieval**: Fast context lookup with caching and inheritance - **Processing**: Real-time context evolution tracking and validation ## Architecture Overview ### Design Principles 1. **Native Go Integration**: Follows established BZZZ patterns for interfaces, error handling, and configuration 2. **Distributed-First**: Designed for P2P environments with role-based access control 3. **Bounded Operations**: Configurable limits prevent excessive resource consumption 4. **Temporal Reasoning**: Tracks decision evolution, not just chronological time 5. **Leader-Only Generation**: Context generation restricted to elected admin nodes 6. **Encryption by Default**: All context data encrypted using existing `pkg/crypto` patterns ### System Components ``` pkg/slurp/ ├── context/ │ ├── resolver.go # Hierarchical context resolution │ ├── hierarchy.go # Bounded hierarchy traversal │ ├── cache.go # Context caching and invalidation │ └── global.go # Global context management ├── temporal/ │ ├── graph.go # Temporal context graph │ ├── evolution.go # Context evolution tracking │ ├── decisions.go # Decision metadata and analysis │ └── navigation.go # Decision-hop navigation ├── storage/ │ ├── distributed.go # DHT-based distributed storage │ ├── encrypted.go # Role-based encrypted storage │ ├── metadata.go # Metadata index management │ └── persistence.go # Local persistence layer ├── intelligence/ │ ├── generator.go # Context generation (admin-only) │ ├── analyzer.go # Context analysis and validation │ ├── patterns.go # Pattern detection and matching │ └── confidence.go # Confidence scoring system ├── retrieval/ │ ├── query.go # Context query interface │ ├── search.go # Search and filtering │ ├── index.go # Search indexing │ └── aggregation.go # Multi-source aggregation └── slurp.go # Main SLURP coordinator ``` ## Core Data Types ### Context Types ```go // ContextNode represents a single context entry in the hierarchy type ContextNode struct { // Identity ID string `json:"id"` UCXLAddress string `json:"ucxl_address"` Path string `json:"path"` // Core Context Summary string `json:"summary"` Purpose string `json:"purpose"` Technologies []string `json:"technologies"` Tags []string `json:"tags"` Insights []string `json:"insights"` // Hierarchy Parent *string `json:"parent,omitempty"` Children []string `json:"children"` Specificity int `json:"specificity"` // Metadata FileType string `json:"file_type"` Language *string `json:"language,omitempty"` Size *int64 `json:"size,omitempty"` LastModified *time.Time `json:"last_modified,omitempty"` ContentHash *string `json:"content_hash,omitempty"` // Resolution CreatedBy string `json:"created_by"` CreatedAt time.Time `json:"created_at"` UpdatedAt time.Time `json:"updated_at"` Confidence float64 `json:"confidence"` // Cascading Rules AppliesTo ContextScope `json:"applies_to"` Overrides bool `json:"overrides"` // Encryption EncryptedFor []string `json:"encrypted_for"` AccessLevel crypto.AccessLevel `json:"access_level"` } // ResolvedContext represents the final resolved context for a UCXL address type ResolvedContext struct { // Resolution Result UCXLAddress string `json:"ucxl_address"` Summary string `json:"summary"` Purpose string `json:"purpose"` Technologies []string `json:"technologies"` Tags []string `json:"tags"` Insights []string `json:"insights"` // Resolution Metadata SourcePath string `json:"source_path"` InheritanceChain []string `json:"inheritance_chain"` Confidence float64 `json:"confidence"` BoundedDepth int `json:"bounded_depth"` GlobalApplied bool `json:"global_applied"` // Temporal Version int `json:"version"` LastUpdated time.Time `json:"last_updated"` EvolutionHistory []string `json:"evolution_history"` // Access Control AccessibleBy []string `json:"accessible_by"` EncryptionKeys []string `json:"encryption_keys"` } type ContextScope string const ( ScopeLocal ContextScope = "local" // Only this file/directory ScopeChildren ContextScope = "children" // This and child directories ScopeGlobal ContextScope = "global" // Entire project ) ``` ### Temporal Types ```go // TemporalNode represents context at a specific decision point type TemporalNode struct { // Identity ID string `json:"id"` UCXLAddress string `json:"ucxl_address"` Version int `json:"version"` // Context Data Context ContextNode `json:"context"` // Temporal Metadata Timestamp time.Time `json:"timestamp"` DecisionID string `json:"decision_id"` ChangeReason ChangeReason `json:"change_reason"` ParentNode *string `json:"parent_node,omitempty"` // Evolution Tracking ContextHash string `json:"context_hash"` Confidence float64 `json:"confidence"` Staleness float64 `json:"staleness"` // Decision Graph Influences []string `json:"influences"` InfluencedBy []string `json:"influenced_by"` // Validation ValidatedBy []string `json:"validated_by"` LastValidated time.Time `json:"last_validated"` } // DecisionMetadata represents metadata about a decision that changed context type DecisionMetadata struct { // Decision Identity ID string `json:"id"` Maker string `json:"maker"` Rationale string `json:"rationale"` // Impact Analysis Scope ImpactScope `json:"scope"` ConfidenceLevel float64 `json:"confidence_level"` // References ExternalRefs []string `json:"external_refs"` GitCommit *string `json:"git_commit,omitempty"` IssueNumber *int `json:"issue_number,omitempty"` // Timing CreatedAt time.Time `json:"created_at"` EffectiveAt *time.Time `json:"effective_at,omitempty"` } type ChangeReason string const ( ReasonInitialCreation ChangeReason = "initial_creation" ReasonCodeChange ChangeReason = "code_change" ReasonDesignDecision ChangeReason = "design_decision" ReasonRefactoring ChangeReason = "refactoring" ReasonArchitectureChange ChangeReason = "architecture_change" ReasonRequirementsChange ChangeReason = "requirements_change" ReasonLearningEvolution ChangeReason = "learning_evolution" ReasonRAGEnhancement ChangeReason = "rag_enhancement" ReasonTeamInput ChangeReason = "team_input" ReasonBugDiscovery ChangeReason = "bug_discovery" ReasonPerformanceInsight ChangeReason = "performance_insight" ReasonSecurityReview ChangeReason = "security_review" ) type ImpactScope string const ( ImpactLocal ImpactScope = "local" ImpactModule ImpactScope = "module" ImpactProject ImpactScope = "project" ImpactSystem ImpactScope = "system" ) ``` ## Core Interfaces ### Context Resolution Interface ```go // ContextResolver defines the interface for hierarchical context resolution type ContextResolver interface { // Resolve resolves context for a UCXL address using cascading inheritance Resolve(ctx context.Context, ucxlAddress string) (*ResolvedContext, error) // ResolveWithDepth resolves context with bounded depth limit ResolveWithDepth(ctx context.Context, ucxlAddress string, maxDepth int) (*ResolvedContext, error) // BatchResolve efficiently resolves multiple UCXL addresses BatchResolve(ctx context.Context, addresses []string) (map[string]*ResolvedContext, error) // InvalidateCache invalidates cached resolution for an address InvalidateCache(ucxlAddress string) error // GetStatistics returns resolver statistics GetStatistics() ResolverStatistics } // HierarchyManager manages the context hierarchy with bounded traversal type HierarchyManager interface { // LoadHierarchy loads the context hierarchy from storage LoadHierarchy(ctx context.Context) error // AddNode adds a context node to the hierarchy AddNode(ctx context.Context, node *ContextNode) error // UpdateNode updates an existing context node UpdateNode(ctx context.Context, node *ContextNode) error // RemoveNode removes a context node and handles children RemoveNode(ctx context.Context, nodeID string) error // TraverseUp traverses up the hierarchy with bounded depth TraverseUp(ctx context.Context, startPath string, maxDepth int) ([]*ContextNode, error) // GetChildren gets immediate children of a node GetChildren(ctx context.Context, nodeID string) ([]*ContextNode, error) // ValidateHierarchy validates hierarchy integrity ValidateHierarchy(ctx context.Context) error } // GlobalContextManager manages global contexts that apply everywhere type GlobalContextManager interface { // AddGlobalContext adds a context that applies globally AddGlobalContext(ctx context.Context, context *ContextNode) error // RemoveGlobalContext removes a global context RemoveGlobalContext(ctx context.Context, contextID string) error // ListGlobalContexts lists all global contexts ListGlobalContexts(ctx context.Context) ([]*ContextNode, error) // ApplyGlobalContexts applies global contexts to a resolution ApplyGlobalContexts(ctx context.Context, resolved *ResolvedContext) error } ``` ### Temporal Analysis Interface ```go // TemporalGraph manages the temporal evolution of context type TemporalGraph interface { // CreateInitialContext creates the first version of context CreateInitialContext(ctx context.Context, ucxlAddress string, contextData *ContextNode, creator string) (*TemporalNode, error) // EvolveContext creates a new temporal version due to a decision EvolveContext(ctx context.Context, ucxlAddress string, newContext *ContextNode, reason ChangeReason, decision *DecisionMetadata) (*TemporalNode, error) // GetLatestVersion gets the most recent temporal node GetLatestVersion(ctx context.Context, ucxlAddress string) (*TemporalNode, error) // GetVersionAtDecision gets context as it was at a specific decision point GetVersionAtDecision(ctx context.Context, ucxlAddress string, decisionHop int) (*TemporalNode, error) // GetEvolutionHistory gets complete evolution history GetEvolutionHistory(ctx context.Context, ucxlAddress string) ([]*TemporalNode, error) // AddInfluenceRelationship adds influence between contexts AddInfluenceRelationship(ctx context.Context, influencer, influenced string) error // FindRelatedDecisions finds decisions within N decision hops FindRelatedDecisions(ctx context.Context, ucxlAddress string, maxHops int) ([]*DecisionPath, error) // FindDecisionPath finds shortest decision path between addresses FindDecisionPath(ctx context.Context, from, to string) ([]*DecisionStep, error) // AnalyzeDecisionPatterns analyzes decision-making patterns AnalyzeDecisionPatterns(ctx context.Context) (*DecisionAnalysis, error) } // DecisionNavigator handles decision-hop based navigation type DecisionNavigator interface { // NavigateDecisionHops navigates by decision distance, not time NavigateDecisionHops(ctx context.Context, ucxlAddress string, hops int, direction NavigationDirection) (*TemporalNode, error) // GetDecisionTimeline gets timeline ordered by decision sequence GetDecisionTimeline(ctx context.Context, ucxlAddress string, includeRelated bool, maxHops int) (*DecisionTimeline, error) // FindStaleContexts finds contexts that may be outdated FindStaleContexts(ctx context.Context, stalenessThreshold float64) ([]*StaleContext, error) // ValidateDecisionPath validates a decision path is reachable ValidateDecisionPath(ctx context.Context, path []*DecisionStep) error } ``` ### Storage Interface ```go // DistributedStorage handles distributed storage of context data type DistributedStorage interface { // Store stores context data in the DHT with encryption Store(ctx context.Context, key string, data interface{}, accessLevel crypto.AccessLevel) error // Retrieve retrieves and decrypts context data Retrieve(ctx context.Context, key string) (interface{}, error) // Delete removes context data from storage Delete(ctx context.Context, key string) error // Index creates searchable indexes for context data Index(ctx context.Context, key string, metadata *IndexMetadata) error // Search searches indexed context data Search(ctx context.Context, query *SearchQuery) ([]*SearchResult, error) // Sync synchronizes with other nodes Sync(ctx context.Context) error } // EncryptedStorage provides role-based encrypted storage type EncryptedStorage interface { // StoreEncrypted stores data encrypted for specific roles StoreEncrypted(ctx context.Context, key string, data interface{}, roles []string) error // RetrieveDecrypted retrieves and decrypts data using current role RetrieveDecrypted(ctx context.Context, key string) (interface{}, error) // CanAccess checks if current role can access data CanAccess(ctx context.Context, key string) (bool, error) // ListAccessibleKeys lists keys accessible to current role ListAccessibleKeys(ctx context.Context) ([]string, error) // ReEncryptForRoles re-encrypts data for different roles ReEncryptForRoles(ctx context.Context, key string, newRoles []string) error } ``` ### Intelligence Interface ```go // ContextGenerator generates context metadata (admin-only) type ContextGenerator interface { // GenerateContext generates context for a path (requires admin role) GenerateContext(ctx context.Context, path string, options *GenerationOptions) (*ContextNode, error) // RegenerateHierarchy regenerates entire hierarchy (admin-only) RegenerateHierarchy(ctx context.Context, rootPath string, options *GenerationOptions) (*HierarchyStats, error) // ValidateGeneration validates generated context quality ValidateGeneration(ctx context.Context, context *ContextNode) (*ValidationResult, error) // EstimateGenerationCost estimates resource cost of generation EstimateGenerationCost(ctx context.Context, scope string) (*CostEstimate, error) } // ContextAnalyzer analyzes context data for patterns and quality type ContextAnalyzer interface { // AnalyzeContext analyzes context quality and consistency AnalyzeContext(ctx context.Context, context *ContextNode) (*AnalysisResult, error) // DetectPatterns detects patterns across contexts DetectPatterns(ctx context.Context, contexts []*ContextNode) ([]*Pattern, error) // SuggestImprovements suggests context improvements SuggestImprovements(ctx context.Context, context *ContextNode) ([]*Suggestion, error) // CalculateConfidence calculates confidence score CalculateConfidence(ctx context.Context, context *ContextNode) (float64, error) // DetectInconsistencies detects inconsistencies in hierarchy DetectInconsistencies(ctx context.Context) ([]*Inconsistency, error) } // PatternMatcher matches context patterns and templates type PatternMatcher interface { // MatchPatterns matches context against known patterns MatchPatterns(ctx context.Context, context *ContextNode) ([]*PatternMatch, error) // RegisterPattern registers a new context pattern RegisterPattern(ctx context.Context, pattern *ContextPattern) error // UnregisterPattern removes a context pattern UnregisterPattern(ctx context.Context, patternID string) error // ListPatterns lists all registered patterns ListPatterns(ctx context.Context) ([]*ContextPattern, error) // UpdatePattern updates an existing pattern UpdatePattern(ctx context.Context, pattern *ContextPattern) error } ``` ## Integration with Existing BZZZ Systems ### DHT Integration ```go // SLURPDHTStorage integrates SLURP with existing DHT type SLURPDHTStorage struct { dht dht.DHT crypto *crypto.AgeCrypto config *config.Config // Context data keys contextPrefix string temporalPrefix string hierarchyPrefix string // Caching cache map[string]interface{} cacheMux sync.RWMutex cacheTTL time.Duration } // Integration points: // - Uses existing pkg/dht for distributed storage // - Leverages dht.DHT.PutValue/GetValue for context data // - Uses dht.DHT.Provide/FindProviders for discovery // - Integrates with dht.DHT peer management ``` ### Crypto Integration ```go // SLURPCrypto extends existing crypto for context-specific needs type SLURPCrypto struct { *crypto.AgeCrypto // SLURP-specific encryption contextRoles map[string][]string // context_type -> allowed_roles defaultRoles []string // default encryption roles } // Integration points: // - Uses existing pkg/crypto/AgeCrypto for role-based encryption // - Extends crypto.AgeCrypto.EncryptForRole for context data // - Uses crypto.AgeCrypto.CanDecryptContent for access control // - Integrates with existing role hierarchy ``` ### Election Integration ```go // SLURPElectionHandler handles election events for admin-only operations type SLURPElectionHandler struct { election *election.ElectionManager slurp *SLURP // Admin-only capabilities canGenerate bool canRegenerate bool canValidate bool } // Integration points: // - Uses existing pkg/election for admin determination // - Only allows context generation when node is admin // - Handles election changes gracefully // - Propagates admin context changes to cluster ``` ### Configuration Integration ```go // SLURP configuration extends existing config.Config type SLURPConfig struct { // Enable/disable SLURP Enabled bool `yaml:"enabled" json:"enabled"` // Context Resolution ContextResolution ContextResolutionConfig `yaml:"context_resolution" json:"context_resolution"` // Temporal Analysis TemporalAnalysis TemporalAnalysisConfig `yaml:"temporal_analysis" json:"temporal_analysis"` // Storage Storage SLURPStorageConfig `yaml:"storage" json:"storage"` // Intelligence Intelligence IntelligenceConfig `yaml:"intelligence" json:"intelligence"` // Performance Performance PerformanceConfig `yaml:"performance" json:"performance"` } // Integration with existing config.SlurpConfig in pkg/config/slurp_config.go ``` ## Concurrency Patterns ### Context Resolution Concurrency ```go // ConcurrentResolver provides thread-safe context resolution type ConcurrentResolver struct { resolver ContextResolver // Concurrency control semaphore chan struct{} // Limit concurrent resolutions cache sync.Map // Thread-safe cache // Request deduplication inflight sync.Map // Deduplicate identical requests // Metrics activeRequests int64 // Atomic counter totalRequests int64 // Atomic counter } // Worker pool pattern for batch operations type ResolverWorkerPool struct { workers int requests chan *ResolveRequest results chan *ResolveResult ctx context.Context cancel context.CancelFunc wg sync.WaitGroup } ``` ### Temporal Graph Concurrency ```go // ConcurrentTemporalGraph provides thread-safe temporal operations type ConcurrentTemporalGraph struct { graph TemporalGraph // Fine-grained locking addressLocks sync.Map // Per-address mutexes // Read-write separation readers sync.RWMutex // Global readers lock // Event-driven updates eventChan chan *TemporalEvent eventWorkers int } ``` ## Performance Optimizations ### Caching Strategy ```go // Multi-level caching for optimal performance type SLURPCache struct { // L1: In-memory cache for frequently accessed contexts l1Cache *ristretto.Cache // L2: Redis cache for shared cluster caching l2Cache redis.UniversalClient // L3: Local disk cache for persistence l3Cache *badger.DB // Cache coordination cacheSync sync.RWMutex metrics *CacheMetrics } ``` ### Bounded Operations ```go // All operations include configurable bounds to prevent resource exhaustion type BoundedOperations struct { MaxDepth int // Hierarchy traversal depth MaxDecisionHops int // Decision graph traversal MaxCacheSize int64 // Memory cache limit MaxConcurrentReqs int // Concurrent resolution limit MaxBatchSize int // Batch operation size RequestTimeout time.Duration // Individual request timeout BackgroundTimeout time.Duration // Background task timeout } ``` ## Error Handling Following BZZZ patterns for consistent error handling: ```go // SLURPError represents SLURP-specific errors type SLURPError struct { Code ErrorCode `json:"code"` Message string `json:"message"` Context map[string]interface{} `json:"context,omitempty"` Cause error `json:"-"` } type ErrorCode string const ( ErrCodeContextNotFound ErrorCode = "context_not_found" ErrCodeDepthLimitExceeded ErrorCode = "depth_limit_exceeded" ErrCodeInvalidUCXL ErrorCode = "invalid_ucxl_address" ErrCodeAccessDenied ErrorCode = "access_denied" ErrCodeTemporalConstraint ErrorCode = "temporal_constraint" ErrCodeGenerationFailed ErrorCode = "generation_failed" ErrCodeStorageError ErrorCode = "storage_error" ErrCodeDecryptionFailed ErrorCode = "decryption_failed" ErrCodeAdminRequired ErrorCode = "admin_required" ErrCodeHierarchyCorrupted ErrorCode = "hierarchy_corrupted" ) ``` ## Implementation Phases ### Phase 1: Foundation (2-3 weeks) 1. **Core Data Types** - Implement all Go structs and interfaces 2. **Basic Context Resolution** - Simple hierarchical resolution 3. **Configuration Integration** - Extend existing config system 4. **Storage Foundation** - Basic encrypted DHT storage ### Phase 2: Hierarchy System (2-3 weeks) 1. **Bounded Hierarchy Walker** - Implement depth-limited traversal 2. **Global Context Support** - System-wide applicable contexts 3. **Caching Layer** - Multi-level caching implementation 4. **Performance Optimization** - Concurrent resolution patterns ### Phase 3: Temporal Intelligence (3-4 weeks) 1. **Temporal Graph** - Decision-based evolution tracking 2. **Decision Navigation** - Decision-hop based traversal 3. **Pattern Analysis** - Context pattern detection 4. **Relationship Mapping** - Influence relationship tracking ### Phase 4: Advanced Features (2-3 weeks) 1. **Context Generation** - Admin-only intelligent generation 2. **Quality Analysis** - Context quality and consistency checking 3. **Search and Indexing** - Advanced context search capabilities 4. **Analytics Dashboard** - Decision pattern visualization ### Phase 5: Integration Testing (1-2 weeks) 1. **End-to-End Testing** - Full BZZZ integration testing 2. **Performance Benchmarking** - Load and stress testing 3. **Security Validation** - Role-based access control testing 4. **Documentation** - Complete API and integration documentation ## Testing Strategy ### Unit Testing - All interfaces mocked using `gomock` - Comprehensive test coverage for core algorithms - Property-based testing for hierarchy operations - Crypto integration testing with test keys ### Integration Testing - DHT integration with mock and real backends - Election integration testing with role changes - Cross-package integration testing - Temporal consistency validation ### Performance Testing - Concurrent resolution benchmarking - Memory usage profiling - Cache effectiveness testing - Bounded operation verification ### Security Testing - Role-based access control validation - Encryption/decryption correctness - Key rotation handling - Attack scenario simulation ## Deployment Considerations ### Configuration Management - Backward-compatible configuration extension - Environment-specific tuning parameters - Feature flags for gradual rollout - Hot configuration reloading ### Monitoring and Observability - Prometheus metrics integration - Structured logging with context - Distributed tracing support - Health check endpoints ### Migration Strategy - Gradual feature enablement - Python-to-Go data migration tools - Fallback mechanisms during transition - Version compatibility matrices ## Conclusion This architecture provides a comprehensive, Go-native implementation of the SLURP contextual intelligence system that integrates seamlessly with existing BZZZ infrastructure. The design emphasizes: - **Native Integration**: Follows established BZZZ patterns and interfaces - **Distributed Architecture**: Built for P2P environments from the ground up - **Security First**: Role-based encryption and access control throughout - **Performance**: Bounded operations and multi-level caching - **Maintainability**: Clear separation of concerns and testable interfaces The phased implementation approach allows for incremental development and testing, ensuring each component integrates properly with the existing BZZZ ecosystem while maintaining system stability and security.