# Phase 2 Hybrid Architecture - BZZZ-RUSTLE Integration

## Overview

Phase 2 introduces a hybrid system where real implementations can be selectively activated while maintaining mock fallbacks. This approach allows gradual transition from mock to production components with zero-downtime deployment and easy rollback capabilities.

## Architecture Principles

### 1. Feature Flag System
- **Environment-based configuration**: Use environment variables and config files
- **Runtime switching**: Components can be switched without recompilation
- **Graceful degradation**: Automatic fallback to mock when real components fail
- **A/B testing**: Support for partial rollouts and testing scenarios

### 2. Interface Compatibility
- **Identical APIs**: Real implementations must match mock interfaces exactly
- **Transparent switching**: Client code unaware of backend implementation
- **Consistent behavior**: Same semantics across mock and real implementations
- **Error handling**: Unified error types and recovery mechanisms

### 3. Deployment Strategy
- **Progressive rollout**: Enable real components incrementally
- **Feature toggles**: Individual component activation control
- **Monitoring integration**: Health checks and performance metrics
- **Rollback capability**: Instant fallback to stable mock components

## Component Architecture

### BZZZ Hybrid Components

#### 1. DHT Backend (Priority 1)
```go
// pkg/dht/hybrid_dht.go
type HybridDHT struct {
    mockDHT    *MockDHT
    realDHT    *LibP2PDHT
    config     *HybridConfig
    fallback   bool
}

type HybridConfig struct {
    UseRealDHT        bool          `env:"BZZZ_USE_REAL_DHT" default:"false"`
    DHTBootstrapNodes []string      `env:"BZZZ_DHT_BOOTSTRAP_NODES"`
    FallbackOnError   bool          `env:"BZZZ_FALLBACK_ON_ERROR" default:"true"`
    HealthCheckInterval time.Duration `env:"BZZZ_HEALTH_CHECK_INTERVAL" default:"30s"`
}
```

**Real Implementation Features**:
- libp2p-based distributed hash table
- Bootstrap node discovery
- Peer-to-peer replication
- Content-addressed storage
- Network partition tolerance

#### 2. UCXL Address Resolution (Priority 2)
```go
// pkg/ucxl/hybrid_resolver.go
type HybridResolver struct {
    localCache  map[string]*UCXLAddress
    dhtResolver *DHTResolver
    config      *ResolverConfig
}

type ResolverConfig struct {
    CacheEnabled    bool          `env:"BZZZ_CACHE_ENABLED" default:"true"`
    CacheTTL        time.Duration `env:"BZZZ_CACHE_TTL" default:"5m"`
    UseDistributed  bool          `env:"BZZZ_USE_DISTRIBUTED_RESOLVER" default:"false"`
}
```

#### 3. Peer Discovery (Priority 3)
```go
// pkg/discovery/hybrid_discovery.go
type HybridDiscovery struct {
    mdns     *MDNSDiscovery
    dht      *DHTDiscovery
    announce *AnnounceDiscovery
    config   *DiscoveryConfig
}
```

### RUSTLE Hybrid Components

#### 1. BZZZ Connector (Priority 1)
```rust
// src/hybrid_bzzz.rs
pub struct HybridBZZZConnector {
    mock_connector: MockBZZZConnector,
    real_connector: Option<RealBZZZConnector>,
    config: HybridConfig,
    health_monitor: HealthMonitor,
}

#[derive(Debug, Clone)]
pub struct HybridConfig {
    pub use_real_connector: bool,
    pub bzzz_endpoints: Vec<String>,
    pub fallback_enabled: bool,
    pub timeout_ms: u64,
    pub retry_attempts: u8,
}
```

#### 2. Network Layer (Priority 2)
```rust
// src/network/hybrid_network.rs
pub struct HybridNetworkLayer {
    mock_network: MockNetwork,
    libp2p_network: Option<LibP2PNetwork>,
    config: NetworkConfig,
}
```

## Feature Flag Implementation

### Environment Configuration
```bash
# BZZZ Configuration
export BZZZ_USE_REAL_DHT=true
export BZZZ_DHT_BOOTSTRAP_NODES="192.168.1.100:8080,192.168.1.101:8080"
export BZZZ_FALLBACK_ON_ERROR=true
export BZZZ_USE_DISTRIBUTED_RESOLVER=false

# RUSTLE Configuration  
export RUSTLE_USE_REAL_CONNECTOR=true
export RUSTLE_BZZZ_ENDPOINTS="http://192.168.1.100:8080,http://192.168.1.101:8080"
export RUSTLE_FALLBACK_ENABLED=true
export RUSTLE_TIMEOUT_MS=5000
```

### Configuration Files
```yaml
# config/hybrid.yaml
bzzz:
  dht:
    enabled: true
    backend: "real"  # mock, real, hybrid
    bootstrap_nodes:
      - "192.168.1.100:8080"
      - "192.168.1.101:8080"
    fallback:
      enabled: true
      threshold_errors: 3
      backoff_ms: 1000

rustle:
  connector:
    enabled: true
    backend: "real"  # mock, real, hybrid
    endpoints:
      - "http://192.168.1.100:8080"
      - "http://192.168.1.101:8080"
    fallback:
      enabled: true
      timeout_ms: 5000
```

## Implementation Phases

### Phase 2.1: Foundation Components (Week 1)
**Priority**: Infrastructure and core interfaces

**BZZZ Tasks**:
1. ✅ Create hybrid DHT interface with feature flags
2. ✅ Implement libp2p-based real DHT backend  
3. ✅ Add health monitoring and fallback logic
4. ✅ Create hybrid configuration system

**RUSTLE Tasks**:
1. ✅ Create hybrid BZZZ connector interface
2. ✅ Implement real HTTP/WebSocket connector
3. ✅ Add connection pooling and retry logic
4. ✅ Create health monitoring system

### Phase 2.2: Service Discovery (Week 2)
**Priority**: Network topology and peer discovery

**BZZZ Tasks**:
1. ✅ Implement mDNS local discovery
2. ✅ Add DHT-based peer discovery
3. ✅ Create announce channel system
4. ✅ Add service capability advertisement

**RUSTLE Tasks**:
1. ✅ Implement service discovery client
2. ✅ Add automatic endpoint resolution
3. ✅ Create connection failover logic
4. ✅ Add load balancing for multiple endpoints

### Phase 2.3: Data Synchronization (Week 3)
**Priority**: Consistent state management

**BZZZ Tasks**:
1. ✅ Implement distributed state synchronization
2. ✅ Add conflict resolution mechanisms
3. ✅ Create eventual consistency guarantees
4. ✅ Add data versioning and merkle trees

**RUSTLE Tasks**:
1. ✅ Implement local caching with invalidation
2. ✅ Add optimistic updates with rollback
3. ✅ Create subscription-based updates
4. ✅ Add offline mode with sync-on-reconnect

## Testing Strategy

### Integration Test Matrix

| Component | Mock | Real | Hybrid | Failure Scenario |
|-----------|------|------|--------|------------------|
| BZZZ DHT | ✅ | ✅ | ✅ | ✅ |
| RUSTLE Connector | ✅ | ✅ | ✅ | ✅ |
| Peer Discovery | ✅ | ✅ | ✅ | ✅ |
| State Sync | ✅ | ✅ | ✅ | ✅ |

### Test Scenarios
1. **Pure Mock**: All components using mock implementations
2. **Pure Real**: All components using real implementations  
3. **Mixed Hybrid**: Some mock, some real components
4. **Fallback Testing**: Real components fail, automatic mock fallback
5. **Recovery Testing**: Real components recover, automatic switch back
6. **Network Partition**: Components handle network splits gracefully
7. **Load Testing**: Performance under realistic traffic patterns

## Monitoring and Observability

### Health Checks
```go
type HealthStatus struct {
    Component   string    `json:"component"`
    Backend     string    `json:"backend"`     // "mock", "real", "hybrid"
    Status      string    `json:"status"`      // "healthy", "degraded", "failed"
    LastCheck   time.Time `json:"last_check"`
    ErrorCount  int       `json:"error_count"`
    Latency     time.Duration `json:"latency_ms"`
}
```

### Metrics Collection
```rust
pub struct HybridMetrics {
    pub mock_requests: u64,
    pub real_requests: u64,
    pub fallback_events: u64,
    pub recovery_events: u64,
    pub avg_latency_mock: Duration,
    pub avg_latency_real: Duration,
    pub error_rate_mock: f64,
    pub error_rate_real: f64,
}
```

### Dashboard Integration
- Component status visualization
- Real-time switching events
- Performance comparisons (mock vs real)
- Error rate tracking and alerting
- Capacity planning metrics

## Deployment Guide

### 1. Pre-deployment Checklist
- [ ] Mock components tested and stable
- [ ] Real implementations ready and tested
- [ ] Configuration files prepared
- [ ] Monitoring dashboards configured
- [ ] Rollback procedures documented

### 2. Deployment Process
```bash
# Phase 2.1: Enable DHT backend only
kubectl set env deployment/bzzz-coordinator BZZZ_USE_REAL_DHT=true
kubectl set env deployment/rustle-browser RUSTLE_USE_REAL_CONNECTOR=false

# Phase 2.2: Enable RUSTLE connector
kubectl set env deployment/rustle-browser RUSTLE_USE_REAL_CONNECTOR=true

# Phase 2.3: Enable full hybrid mode
kubectl apply -f config/phase2-hybrid.yaml
```

### 3. Rollback Procedure
```bash
# Emergency rollback to full mock mode
kubectl set env deployment/bzzz-coordinator BZZZ_USE_REAL_DHT=false
kubectl set env deployment/rustle-browser RUSTLE_USE_REAL_CONNECTOR=false
```

## Success Criteria

### Phase 2 Completion Requirements
1. **All Phase 1 tests pass** with hybrid components
2. **Real component integration** working end-to-end
3. **Automatic fallback** triggered and recovered under failure conditions
4. **Performance parity** between mock and real implementations
5. **Zero-downtime switching** between backends validated
6. **Production monitoring** integrated and alerting functional

### Performance Benchmarks
- **DHT Operations**: Real implementation within 2x of mock latency
- **RUSTLE Queries**: End-to-end response time < 500ms
- **Fallback Time**: Mock fallback activated within 100ms of failure detection
- **Recovery Time**: Real backend reactivation within 30s of health restoration

### Reliability Targets
- **Uptime**: 99.9% availability during Phase 2
- **Error Rate**: < 0.1% for hybrid operations
- **Data Consistency**: Zero data loss during backend switching
- **Fallback Success**: 100% successful fallback to mock on real component failure

## Risk Mitigation

### Identified Risks
1. **Real component instability**: Mitigated by automatic fallback
2. **Configuration drift**: Mitigated by infrastructure as code
3. **Performance degradation**: Mitigated by continuous monitoring
4. **Data inconsistency**: Mitigated by transactional operations
5. **Network partitions**: Mitigated by eventual consistency design

### Contingency Plans
- **Immediate rollback** to Phase 1 mock-only mode
- **Component isolation** to contain failures
- **Manual override** for critical operations
- **Emergency contact procedures** for escalation

## Next Steps to Phase 3

Phase 3 preparation begins once Phase 2 stability is achieved:
1. **Remove mock components** from production code paths
2. **Optimize real implementations** for production scale
3. **Add security layers** (encryption, authentication, authorization)
4. **Implement advanced features** (sharding, consensus, Byzantine fault tolerance)
5. **Production hardening** (security audits, penetration testing, compliance)