Major updates and improvements to BZZZ system

- Updated configuration and deployment files - Improved system architecture and components - Enhanced documentation and testing - Fixed various issues and added new features 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-17 18:06:57 +10:00
parent 4e6140de03
commit f5f96ba505
71 changed files with 664 additions and 3823 deletions
--- a/deployments/bare-metal/WEB_UI_DEVELOPMENT_PLAN.md
+++ b/deployments/bare-metal/WEB_UI_DEVELOPMENT_PLAN.md
@@ -0,0 +1,425 @@
+# BZZZ Web Installation/Configuration Development Plan
+
+## Overview
+
+This plan leverages existing BZZZ infrastructure to implement the missing web-based installation and configuration functionality without reinventing the wheel. We'll integrate the existing config-ui with our enhanced installer and BZZZ's existing systems.
+
+## Existing Infrastructure Analysis
+
+### ✅ **What We Already Have**
+
+1. **HTTP API Server** (`api/http_server.go`)
+   - Existing HTTP server with CORS support
+   - Health endpoints (`/api/health`, `/api/status`)
+   - Hypercore log API endpoints
+   - Gorilla Mux router setup
+
+2. **Config UI Foundation** (`install/config-ui/`)
+   - Complete Next.js 14 setup
+   - 8-step setup wizard framework
+   - TypeScript + Tailwind CSS
+   - Component structure already defined
+   - Progress tracking and navigation
+
+3. **Task Coordinator** (`coordinator/task_coordinator.go`)
+   - Agent management and role assignment
+   - Repository integration framework
+   - Status reporting capabilities
+
+4. **Configuration System** (`pkg/config/`)
+   - YAML configuration management
+   - Role and agent configuration
+   - Network and service settings
+
+5. **Repository Integration** (`repository/factory.go`)
+   - GITEA and GitHub providers
+   - Task management interfaces
+   - Credential handling
+
+## Implementation Strategy
+
+### Phase 1: Backend API Integration (1-2 days)
+
+#### 1.1 Extend HTTP Server with Setup APIs
+**File**: `api/http_server.go`
+
+Add new endpoints to existing server:
+```go
+// Setup and configuration endpoints
+api.HandleFunc("/setup/system", h.handleSystemInfo).Methods("GET")
+api.HandleFunc("/setup/network", h.handleNetworkConfig).Methods("GET", "POST")
+api.HandleFunc("/setup/security", h.handleSecurityConfig).Methods("GET", "POST")
+api.HandleFunc("/setup/ai", h.handleAIConfig).Methods("GET", "POST")
+api.HandleFunc("/setup/resources", h.handleResourceConfig).Methods("GET", "POST")
+api.HandleFunc("/setup/deploy", h.handleServiceDeploy).Methods("POST")
+api.HandleFunc("/setup/cluster", h.handleClusterConfig).Methods("GET", "POST")
+api.HandleFunc("/setup/validate", h.handleValidation).Methods("POST")
+
+// Repository configuration (integrate with existing factory)
+api.HandleFunc("/setup/repository", h.handleRepositoryConfig).Methods("GET", "POST")
+```
+
+#### 1.2 System Detection Integration
+Leverage existing system info from enhanced installer:
+```go
+func (h *HTTPServer) handleSystemInfo(w http.ResponseWriter, r *http.Request) {
+    info := map[string]interface{}{
+        "os":           detectOS(),           // From enhanced installer
+        "architecture": detectArchitecture(), // From enhanced installer  
+        "hardware":     detectHardware(),     // New GPU detection
+        "network":      detectNetwork(),      // Network interface discovery
+        "services":     detectServices(),     // Docker, Ollama status
+    }
+    json.NewEncoder(w).Encode(info)
+}
+```
+
+#### 1.3 Configuration Management
+Extend existing config system to support web updates:
+```go
+// New file: api/setup_handlers.go
+type SetupManager struct {
+    configPath   string
+    coordinator  *coordinator.TaskCoordinator
+    repoFactory  *repository.DefaultProviderFactory
+}
+
+func (s *SetupManager) UpdateNetworkConfig(config NetworkConfig) error {
+    // Update YAML configuration
+    // Restart network services if needed
+    // Validate connectivity
+}
+```
+
+### Phase 2: Web UI Enhancement (2-3 days)
+
+#### 2.1 Complete Existing Components
+The config-ui framework exists but components need implementation:
+
+**SystemDetection.tsx** - Leverage system detection API:
+```typescript
+const SystemDetection = ({ onComplete, systemInfo }) => {
+    // Display detected hardware (GPU detection for Parallama)
+    // Show system requirements validation
+    // Network interface selection
+    // Prerequisite checking
+}
+```
+
+**NetworkConfiguration.tsx** - Network setup:
+```typescript
+const NetworkConfiguration = ({ onComplete, configData }) => {
+    // Port configuration (8080, 8081, 4001)
+    // Firewall rules setup
+    // Network interface selection
+    // Connectivity testing
+}
+```
+
+**SecuritySetup.tsx** - Security configuration:
+```typescript
+const SecuritySetup = ({ onComplete, configData }) => {
+    // SSH key generation/upload (for cluster deployment)
+    // TLS certificate setup
+    // Authentication method selection
+    // Security policy configuration
+}
+```
+
+#### 2.2 Repository Integration Component
+**File**: `install/config-ui/app/setup/components/RepositoryConfiguration.tsx`
+```typescript
+const RepositoryConfiguration = ({ onComplete, configData }) => {
+    // GITEA/GitHub provider selection
+    // Credential input and validation
+    // Repository access testing
+    // Task label configuration
+    // Integration with existing repository factory
+}
+```
+
+#### 2.3 AI Configuration Enhancement
+**AIConfiguration.tsx** - GPU optimization:
+```typescript
+const AIConfiguration = ({ onComplete, systemInfo }) => {
+    // GPU detection display
+    // Parallama vs Ollama recommendation
+    // OpenAI API configuration
+    // Model selection and downloading
+    // Resource allocation per model
+}
+```
+
+### Phase 3: Installer Integration (1 day)
+
+#### 3.1 Enhanced Installer Web Mode
+**File**: `/home/tony/chorus/project-queues/active/chorus.services/installer/install-chorus-enhanced.sh`
+
+Add web UI mode option:
+```bash
+# Add new command line option
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --web-ui)
+            ENABLE_WEB_UI=true
+            shift
+            ;;
+        # ... existing options
+    esac
+done
+
+# After basic installation
+if [[ "$ENABLE_WEB_UI" == "true" ]]; then
+    setup_web_ui
+fi
+
+setup_web_ui() {
+    log_step "Setting up web configuration interface..."
+    
+    # Copy config-ui to BZZZ directory
+    cp -r "$BZZZ_DIR/../install/config-ui" "$BZZZ_DIR/web-ui"
+    
+    # Install Node.js and dependencies
+    install_nodejs
+    cd "$BZZZ_DIR/web-ui" && npm install
+    
+    # Start web UI in background
+    npm run build && npm run start &
+    
+    echo ""
+    echo "🌐 Web configuration available at: http://$(hostname):8080/setup"
+    echo "⚡ Continue setup in your browser"
+}
+```
+
+#### 3.2 Hybrid Installation Flow
+```bash
+# Enhanced installer usage options:
+curl -fsSL https://chorus.services/install-enhanced.sh | sh                    # CLI mode (current)
+curl -fsSL https://chorus.services/install-enhanced.sh | sh -s -- --web-ui    # Web mode (new)
+```
+
+### Phase 4: Cluster Deployment Integration (2-3 days)
+
+#### 4.1 SSH Deployment System
+**File**: `api/cluster_deployment.go`
+```go
+type ClusterDeployer struct {
+    sshConfig   SSHConfig
+    installer   string  // Enhanced installer script
+    coordinator *coordinator.TaskCoordinator
+}
+
+func (c *ClusterDeployer) DeployToNodes(nodes []NodeConfig) error {
+    for _, node := range nodes {
+        // SSH to remote node
+        // Execute enhanced installer with node-specific config
+        // Verify installation
+        // Add to cluster coordination
+    }
+}
+```
+
+#### 4.2 Cluster Formation Component
+**ClusterFormation.tsx** - Multi-node coordination:
+```typescript
+const ClusterFormation = ({ onComplete, configData }) => {
+    // Node discovery (SSH-based or manual IP entry)
+    // SSH credential configuration
+    // Remote deployment progress tracking
+    // Cluster validation and health checking
+    // P2P network formation verification
+}
+```
+
+#### 4.3 P2P Network Integration
+Leverage existing P2P infrastructure:
+```go
+// Integration with existing p2p/node.go
+func (h *HTTPServer) handleClusterConfig(w http.ResponseWriter, r *http.Request) {
+    // Use existing P2P node configuration
+    // Coordinate with task coordinator
+    // Enable automatic peer discovery
+}
+```
+
+### Phase 5: Professional Installation Experience (1-2 days)
+
+#### 5.1 Enhanced Installation Output
+```bash
+# Professional branded installer output
+print_professional_banner() {
+    echo -e "${PURPLE}"
+    cat << 'EOF'
+    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+    🔥 BZZZ Distributed AI Coordination Platform
+       Professional Installation System v2.0
+    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+EOF
+    echo -e "${NC}"
+}
+
+show_installation_progress() {
+    local step="$1"
+    local total="$2"
+    local message="$3"
+    
+    echo -e "${BLUE}[$step/$total]${NC} $message"
+    
+    # Progress bar
+    local progress=$((step * 100 / total))
+    printf "Progress: ["
+    for ((i=0; i<progress/5; i++)); do printf "█"; done
+    for ((i=progress/5; i<20; i++)); do printf "░"; done
+    printf "] %d%%\n" $progress
+}
+```
+
+#### 5.2 GPU Detection and Parallama Integration
+```bash
+detect_gpu_configuration() {
+    log_step "Analyzing GPU configuration..."
+    
+    # Detect NVIDIA GPUs
+    if command -v nvidia-smi >/dev/null 2>&1; then
+        GPU_COUNT=$(nvidia-smi --list-gpus | wc -l)
+        GPU_INFO=$(nvidia-smi --query-gpu=name,memory.total --format=csv,noheader)
+        
+        if [[ $GPU_COUNT -gt 1 ]]; then
+            echo ""
+            echo -e "${GREEN}🚀 Multi-GPU Setup Detected ($GPU_COUNT NVIDIA GPUs)${NC}"
+            echo -e "${CYAN}Parallama is RECOMMENDED for optimal multi-GPU performance!${NC}"
+            echo ""
+            echo "Detected GPUs:"
+            echo "$GPU_INFO" | sed 's/^/  • /'
+            echo ""
+            
+            read -p "Install Parallama (recommended) or standard Ollama? [P/o]: " choice
+            case $choice in
+                [Oo]* ) INSTALL_OLLAMA_TYPE="standard" ;;
+                * ) INSTALL_OLLAMA_TYPE="parallama" ;;
+            esac
+        fi
+    fi
+}
+```
+
+## Implementation Timeline
+
+### Week 1: Backend Foundation
+- **Day 1-2**: Extend HTTP server with setup APIs
+- **Day 3-4**: Implement system detection and configuration management
+- **Day 5**: Repository integration and credential handling
+
+### Week 2: Web UI Development  
+- **Day 1-2**: Complete existing setup components
+- **Day 3-4**: Repository configuration and AI setup components
+- **Day 5**: Integration testing and UI polish
+
+### Week 3: Cluster and Professional Features
+- **Day 1-2**: SSH deployment system and cluster formation
+- **Day 3-4**: Professional installation experience and GPU detection
+- **Day 5**: End-to-end testing and documentation
+
+## Leveraging Existing BZZZ Systems
+
+### ✅ **Reuse Without Modification**
+1. **HTTP Server Architecture** - Extend existing `api/http_server.go`
+2. **Configuration System** - Use existing `pkg/config/` YAML management
+3. **Repository Integration** - Leverage `repository/factory.go` providers
+4. **Task Coordination** - Integrate with `coordinator/task_coordinator.go`
+5. **P2P Networking** - Use existing `p2p/node.go` infrastructure
+
+### 🔧 **Extend Existing Systems**
+1. **Enhanced Installer** - Add web UI mode to existing script
+2. **Config UI Framework** - Complete existing component implementations
+3. **API Endpoints** - Add setup endpoints to existing HTTP server
+4. **System Detection** - Enhance existing OS detection with hardware profiling
+
+### 🆕 **New Components Needed**
+1. **Cluster Deployment Manager** - SSH-based remote installation
+2. **GPU Detection System** - Hardware profiling and Parallama integration
+3. **Professional Installation UX** - Enhanced progress and branding
+4. **Setup API Handlers** - Backend logic for web configuration
+
+## Integration Points
+
+### Repository Integration
+```go
+// Leverage existing repository factory
+func (h *HTTPServer) handleRepositoryConfig(w http.ResponseWriter, r *http.Request) {
+    factory := &repository.DefaultProviderFactory{}
+    
+    // Create provider based on web UI input
+    provider, err := factory.CreateProvider(ctx, repoConfig)
+    
+    // Test connectivity
+    // Store configuration
+    // Update task coordinator
+}
+```
+
+### Task Coordinator Integration
+```go
+// Use existing task coordinator for cluster management
+func (h *HTTPServer) handleClusterStatus(w http.ResponseWriter, r *http.Request) {
+    status := h.taskCoordinator.GetStatus()
+    json.NewEncoder(w).Encode(status)
+}
+```
+
+### Configuration Management
+```go
+// Extend existing config system
+func (s *SetupManager) SaveConfiguration(config SetupConfig) error {
+    // Convert web config to BZZZ YAML format
+    // Use existing config.Config struct
+    // Restart services as needed
+}
+```
+
+## Success Metrics
+
+### Technical Completeness
+- [ ] Web UI accessible at `:8080/setup` 
+- [ ] 8-step configuration wizard functional
+- [ ] GPU detection and Parallama recommendation
+- [ ] SSH-based cluster deployment
+- [ ] Repository integration (GITEA/GitHub)
+- [ ] Professional installation experience
+
+### User Experience
+- [ ] Single-command installation with web option
+- [ ] Intuitive progress tracking and navigation
+- [ ] Real-time validation and error handling
+- [ ] Mobile-responsive design
+- [ ] Comprehensive help and documentation
+
+### Integration Quality
+- [ ] Seamless integration with existing BZZZ systems
+- [ ] No disruption to current enhanced installer
+- [ ] Proper error handling and rollback
+- [ ] Production-ready security and performance
+
+## Risk Mitigation
+
+### Technical Risks
+- **API Integration Complexity**: Use existing HTTP server patterns
+- **Configuration Conflicts**: Maintain YAML compatibility
+- **Service Coordination**: Leverage existing task coordinator
+
+### User Experience Risks  
+- **Installation Complexity**: Provide both CLI and web options
+- **Error Recovery**: Implement proper rollback mechanisms
+- **Performance**: Optimize for low-resource environments
+
+## Conclusion
+
+This plan leverages 80% of existing BZZZ infrastructure while delivering the professional web-based installation experience envisioned in the original plans. By extending rather than replacing existing systems, we minimize development time and maintain compatibility with current deployments.
+
+**Key Benefits:**
+- ✅ Rapid implementation using existing code
+- ✅ Maintains backward compatibility
+- ✅ Professional installation experience
+- ✅ Complete feature parity with original vision
+- ✅ Seamless integration with enhanced installer