bzzz/archive/PHASE2_HYBRID_ARCHITECTURE.md

Phase 2 Hybrid Architecture - BZZZ-RUSTLE Integration

Overview

Phase 2 introduces a hybrid system where real implementations can be selectively activated while maintaining mock fallbacks. This approach allows gradual transition from mock to production components with zero-downtime deployment and easy rollback capabilities.

Architecture Principles

1. Feature Flag System

• Environment-based configuration: Use environment variables and config files
• Runtime switching: Components can be switched without recompilation
• Graceful degradation: Automatic fallback to mock when real components fail
• A/B testing: Support for partial rollouts and testing scenarios

2. Interface Compatibility

• Identical APIs: Real implementations must match mock interfaces exactly
• Transparent switching: Client code unaware of backend implementation
• Consistent behavior: Same semantics across mock and real implementations
• Error handling: Unified error types and recovery mechanisms

3. Deployment Strategy

• Progressive rollout: Enable real components incrementally
• Feature toggles: Individual component activation control
• Monitoring integration: Health checks and performance metrics
• Rollback capability: Instant fallback to stable mock components

Component Architecture

BZZZ Hybrid Components

1. DHT Backend (Priority 1)

// pkg/dht/hybrid_dht.go
type HybridDHT struct {
    mockDHT    *MockDHT
    realDHT    *LibP2PDHT
    config     *HybridConfig
    fallback   bool
}

type HybridConfig struct {
    UseRealDHT        bool          `env:"BZZZ_USE_REAL_DHT" default:"false"`
    DHTBootstrapNodes []string      `env:"BZZZ_DHT_BOOTSTRAP_NODES"`
    FallbackOnError   bool          `env:"BZZZ_FALLBACK_ON_ERROR" default:"true"`
    HealthCheckInterval time.Duration `env:"BZZZ_HEALTH_CHECK_INTERVAL" default:"30s"`
}

Real Implementation Features:

• libp2p-based distributed hash table
• Bootstrap node discovery
• Peer-to-peer replication
• Content-addressed storage
• Network partition tolerance

2. UCXL Address Resolution (Priority 2)

// pkg/ucxl/hybrid_resolver.go
type HybridResolver struct {
    localCache  map[string]*UCXLAddress
    dhtResolver *DHTResolver
    config      *ResolverConfig
}

type ResolverConfig struct {
    CacheEnabled    bool          `env:"BZZZ_CACHE_ENABLED" default:"true"`
    CacheTTL        time.Duration `env:"BZZZ_CACHE_TTL" default:"5m"`
    UseDistributed  bool          `env:"BZZZ_USE_DISTRIBUTED_RESOLVER" default:"false"`
}

3. Peer Discovery (Priority 3)

// pkg/discovery/hybrid_discovery.go
type HybridDiscovery struct {
    mdns     *MDNSDiscovery
    dht      *DHTDiscovery
    announce *AnnounceDiscovery
    config   *DiscoveryConfig
}

RUSTLE Hybrid Components

1. BZZZ Connector (Priority 1)

// src/hybrid_bzzz.rs
pub struct HybridBZZZConnector {
    mock_connector: MockBZZZConnector,
    real_connector: Option<RealBZZZConnector>,
    config: HybridConfig,
    health_monitor: HealthMonitor,
}

#[derive(Debug, Clone)]
pub struct HybridConfig {
    pub use_real_connector: bool,
    pub bzzz_endpoints: Vec<String>,
    pub fallback_enabled: bool,
    pub timeout_ms: u64,
    pub retry_attempts: u8,
}

2. Network Layer (Priority 2)

// src/network/hybrid_network.rs
pub struct HybridNetworkLayer {
    mock_network: MockNetwork,
    libp2p_network: Option<LibP2PNetwork>,
    config: NetworkConfig,
}

Feature Flag Implementation

Environment Configuration

# BZZZ Configuration
export BZZZ_USE_REAL_DHT=true
export BZZZ_DHT_BOOTSTRAP_NODES="192.168.1.100:8080,192.168.1.101:8080"
export BZZZ_FALLBACK_ON_ERROR=true
export BZZZ_USE_DISTRIBUTED_RESOLVER=false

# RUSTLE Configuration  
export RUSTLE_USE_REAL_CONNECTOR=true
export RUSTLE_BZZZ_ENDPOINTS="http://192.168.1.100:8080,http://192.168.1.101:8080"
export RUSTLE_FALLBACK_ENABLED=true
export RUSTLE_TIMEOUT_MS=5000

Configuration Files

# config/hybrid.yaml
bzzz:
  dht:
    enabled: true
    backend: "real"  # mock, real, hybrid
    bootstrap_nodes:
      - "192.168.1.100:8080"
      - "192.168.1.101:8080"
    fallback:
      enabled: true
      threshold_errors: 3
      backoff_ms: 1000

rustle:
  connector:
    enabled: true
    backend: "real"  # mock, real, hybrid
    endpoints:
      - "http://192.168.1.100:8080"
      - "http://192.168.1.101:8080"
    fallback:
      enabled: true
      timeout_ms: 5000

Implementation Phases

Phase 2.1: Foundation Components (Week 1)

Priority: Infrastructure and core interfaces

BZZZ Tasks:

1. ✅ Create hybrid DHT interface with feature flags
2. ✅ Implement libp2p-based real DHT backend
3. ✅ Add health monitoring and fallback logic
4. ✅ Create hybrid configuration system

RUSTLE Tasks:

1. ✅ Create hybrid BZZZ connector interface
2. ✅ Implement real HTTP/WebSocket connector
3. ✅ Add connection pooling and retry logic
4. ✅ Create health monitoring system

Phase 2.2: Service Discovery (Week 2)

Priority: Network topology and peer discovery

BZZZ Tasks:

1. ✅ Implement mDNS local discovery
2. ✅ Add DHT-based peer discovery
3. ✅ Create announce channel system
4. ✅ Add service capability advertisement

RUSTLE Tasks:

1. ✅ Implement service discovery client
2. ✅ Add automatic endpoint resolution
3. ✅ Create connection failover logic
4. ✅ Add load balancing for multiple endpoints

Phase 2.3: Data Synchronization (Week 3)

Priority: Consistent state management

BZZZ Tasks:

1. ✅ Implement distributed state synchronization
2. ✅ Add conflict resolution mechanisms
3. ✅ Create eventual consistency guarantees
4. ✅ Add data versioning and merkle trees

RUSTLE Tasks:

1. ✅ Implement local caching with invalidation
2. ✅ Add optimistic updates with rollback
3. ✅ Create subscription-based updates
4. ✅ Add offline mode with sync-on-reconnect

Testing Strategy

Integration Test Matrix

Component	Mock	Real	Hybrid	Failure Scenario
BZZZ DHT	✅	✅	✅	✅
RUSTLE Connector	✅	✅	✅	✅
Peer Discovery	✅	✅	✅	✅
State Sync	✅	✅	✅	✅

Test Scenarios

1. Pure Mock: All components using mock implementations
2. Pure Real: All components using real implementations
3. Mixed Hybrid: Some mock, some real components
4. Fallback Testing: Real components fail, automatic mock fallback
5. Recovery Testing: Real components recover, automatic switch back
6. Network Partition: Components handle network splits gracefully
7. Load Testing: Performance under realistic traffic patterns

Monitoring and Observability

Health Checks

type HealthStatus struct {
    Component   string    `json:"component"`
    Backend     string    `json:"backend"`     // "mock", "real", "hybrid"
    Status      string    `json:"status"`      // "healthy", "degraded", "failed"
    LastCheck   time.Time `json:"last_check"`
    ErrorCount  int       `json:"error_count"`
    Latency     time.Duration `json:"latency_ms"`
}

Metrics Collection

pub struct HybridMetrics {
    pub mock_requests: u64,
    pub real_requests: u64,
    pub fallback_events: u64,
    pub recovery_events: u64,
    pub avg_latency_mock: Duration,
    pub avg_latency_real: Duration,
    pub error_rate_mock: f64,
    pub error_rate_real: f64,
}

Dashboard Integration

• Component status visualization
• Real-time switching events
• Performance comparisons (mock vs real)
• Error rate tracking and alerting
• Capacity planning metrics

Deployment Guide

1. Pre-deployment Checklist

• Mock components tested and stable
• Real implementations ready and tested
• Configuration files prepared
• Monitoring dashboards configured
• Rollback procedures documented

2. Deployment Process

# Phase 2.1: Enable DHT backend only
kubectl set env deployment/bzzz-coordinator BZZZ_USE_REAL_DHT=true
kubectl set env deployment/rustle-browser RUSTLE_USE_REAL_CONNECTOR=false

# Phase 2.2: Enable RUSTLE connector
kubectl set env deployment/rustle-browser RUSTLE_USE_REAL_CONNECTOR=true

# Phase 2.3: Enable full hybrid mode
kubectl apply -f config/phase2-hybrid.yaml

3. Rollback Procedure

# Emergency rollback to full mock mode
kubectl set env deployment/bzzz-coordinator BZZZ_USE_REAL_DHT=false
kubectl set env deployment/rustle-browser RUSTLE_USE_REAL_CONNECTOR=false

Success Criteria

Phase 2 Completion Requirements

1. All Phase 1 tests pass with hybrid components
2. Real component integration working end-to-end
3. Automatic fallback triggered and recovered under failure conditions
4. Performance parity between mock and real implementations
5. Zero-downtime switching between backends validated
6. Production monitoring integrated and alerting functional

Performance Benchmarks

• DHT Operations: Real implementation within 2x of mock latency
• RUSTLE Queries: End-to-end response time < 500ms
• Fallback Time: Mock fallback activated within 100ms of failure detection
• Recovery Time: Real backend reactivation within 30s of health restoration

Reliability Targets

• Uptime: 99.9% availability during Phase 2
• Error Rate: < 0.1% for hybrid operations
• Data Consistency: Zero data loss during backend switching
• Fallback Success: 100% successful fallback to mock on real component failure

Risk Mitigation

Identified Risks

1. Real component instability: Mitigated by automatic fallback
2. Configuration drift: Mitigated by infrastructure as code
3. Performance degradation: Mitigated by continuous monitoring
4. Data inconsistency: Mitigated by transactional operations
5. Network partitions: Mitigated by eventual consistency design

Contingency Plans

• Immediate rollback to Phase 1 mock-only mode
• Component isolation to contain failures
• Manual override for critical operations
• Emergency contact procedures for escalation

Next Steps to Phase 3

Phase 3 preparation begins once Phase 2 stability is achieved:

1. Remove mock components from production code paths
2. Optimize real implementations for production scale
3. Add security layers (encryption, authentication, authorization)
4. Implement advanced features (sharding, consensus, Byzantine fault tolerance)
5. Production hardening (security audits, penetration testing, compliance)