Commit Graph

5 Commits

Author SHA1 Message Date
anthonyrawlins
e523c4b543 feat: Implement CHORUS scaling improvements for robust autoscaling
Address WHOOSH issue #7 with comprehensive scaling optimizations to prevent
license server, bootstrap peer, and control plane collapse during fast scale-out.

HIGH-RISK FIXES (Must-Do):
 License gate already implemented with cache + circuit breaker + grace window
 mDNS disabled in container environments (CHORUS_MDNS_ENABLED=false)
 Connection rate limiting (5 dials/sec, 16 concurrent DHT queries)
 Connection manager with watermarks (32 low, 128 high)
 AutoNAT enabled for container networking

MEDIUM-RISK FIXES (Next Priority):
 Assignment merge layer with HTTP/file config + SIGHUP reload
 Runtime configuration system with WHOOSH assignment API support
 Election stability windows to prevent churn:
  - CHORUS_ELECTION_MIN_TERM=30s (minimum time between elections)
  - CHORUS_LEADER_MIN_TERM=45s (minimum time before challenging healthy leader)
 Bootstrap pool JSON support with priority sorting and join stagger

NEW FEATURES:
- Runtime config system with assignment overrides from WHOOSH
- SIGHUP reload handler for live configuration updates
- JSON bootstrap configuration with peer metadata (region, roles, priority)
- Configurable election stability windows with environment variables
- Multi-format bootstrap support: Assignment → JSON → CSV

FILES MODIFIED:
- pkg/config/assignment.go (NEW): Runtime assignment merge system
- docker/bootstrap.json (NEW): Example JSON bootstrap configuration
- pkg/election/election.go: Added stability windows and churn prevention
- internal/runtime/shared.go: Integrated assignment loading and conditional mDNS
- p2p/node.go: Added connection management and rate limiting
- pkg/config/hybrid_config.go: Added rate limiting configuration fields
- docker/docker-compose.yml: Updated environment variables and configs
- README.md: Updated status table with scaling milestone

This implementation enables wave-based autoscaling without system collapse,
addressing all scaling concerns from WHOOSH issue #7.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-23 17:50:40 +10:00
anthonyrawlins
26e4ef7d8b feat: Implement complete CHORUS leader election system
Major milestone: CHORUS leader election is now fully functional!

## Key Features Implemented:

### 🗳️ Leader Election Core
- Fixed root cause: nodes now trigger elections when no admin exists
- Added randomized election delays to prevent simultaneous elections
- Implemented concurrent election prevention (only one election at a time)
- Added proper election state management and transitions

### 📡 Admin Discovery System
- Enhanced discovery requests with "WHOAMI" debug messages
- Fixed discovery responses to properly include current leader ID
- Added comprehensive discovery request/response logging
- Implemented admin confirmation from multiple sources

### 🔧 Configuration Improvements
- Increased discovery timeout from 3s to 15s for better reliability
- Added proper Docker Hub image deployment workflow
- Updated build process to use correct chorus-agent binary (not deprecated chorus)
- Added static compilation flags for Alpine Linux compatibility

### 🐛 Critical Fixes
- Fixed build process confusion between chorus vs chorus-agent binaries
- Added missing admin_election capability to enable leader elections
- Corrected discovery logic to handle zero admin responses
- Enhanced debugging with detailed state and timing information

## Current Operational Status:
 Admin Election: Working with proper consensus
 Heartbeat System: 15-second intervals from elected admin
 Discovery Protocol: Nodes can find and confirm current admin
 P2P Connectivity: 5+ connected peers with libp2p
 SLURP Functionality: Enabled on admin nodes
 BACKBEAT Integration: Tempo synchronization working
 Container Health: All health checks passing

## Technical Details:
- Election uses weighted scoring based on uptime, capabilities, and resources
- Randomized delays prevent election storms (30-45s wait periods)
- Discovery responses include current leader ID for network-wide consensus
- State management prevents multiple concurrent elections
- Enhanced logging provides full visibility into election process

🎉 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-23 13:06:53 +10:00
anthonyrawlins
1bb736c09a Harden CHORUS security and messaging stack 2025-09-20 23:21:35 +10:00
anthonyrawlins
9bdcbe0447 Integrate BACKBEAT SDK and resolve KACHING license validation
Major integrations and fixes:
- Added BACKBEAT SDK integration for P2P operation timing
- Implemented beat-aware status tracking for distributed operations
- Added Docker secrets support for secure license management
- Resolved KACHING license validation via HTTPS/TLS
- Updated docker-compose configuration for clean stack deployment
- Disabled rollback policies to prevent deployment failures
- Added license credential storage (CHORUS-DEV-MULTI-001)

Technical improvements:
- BACKBEAT P2P operation tracking with phase management
- Enhanced configuration system with file-based secrets
- Improved error handling for license validation
- Clean separation of KACHING and CHORUS deployment stacks

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-06 07:56:26 +10:00
anthonyrawlins
543ab216f9 Complete BZZZ functionality port to CHORUS
🎭 CHORUS now contains full BZZZ functionality adapted for containers

Core systems ported:
- P2P networking (libp2p with DHT and PubSub)
- Task coordination (COOEE protocol)
- HMMM collaborative reasoning
- SHHH encryption and security
- SLURP admin election system
- UCXL content addressing
- UCXI server integration
- Hypercore logging system
- Health monitoring and graceful shutdown
- License validation with KACHING

Container adaptations:
- Environment variable configuration (no YAML files)
- Container-optimized logging to stdout/stderr
- Auto-generated agent IDs for container deployments
- Docker-first architecture

All proven BZZZ P2P protocols, AI integration, and collaboration
features are now available in containerized form.

Next: Build and test container deployment.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-02 20:02:37 +10:00