Compare commits
2 Commits
e523c4b543
...
feature/re
| Author | SHA1 | Date | |
|---|---|---|---|
| c30c6dc480 | |||
| ef4bf1efe0 |
35
README.md
35
README.md
@@ -8,7 +8,7 @@ CHORUS is the runtime that ties the CHORUS ecosystem together: libp2p mesh, DHT-
|
|||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| libp2p node + PubSub | ✅ Running | `internal/runtime/shared.go` spins up the mesh, hypercore logging, availability broadcasts. |
|
| libp2p node + PubSub | ✅ Running | `internal/runtime/shared.go` spins up the mesh, hypercore logging, availability broadcasts. |
|
||||||
| DHT + DecisionPublisher | ✅ Running | Encrypted storage wired through `pkg/dht`; decisions written via `ucxl.DecisionPublisher`. |
|
| DHT + DecisionPublisher | ✅ Running | Encrypted storage wired through `pkg/dht`; decisions written via `ucxl.DecisionPublisher`. |
|
||||||
| **Leader Election System** | ✅ **FULLY FUNCTIONAL** | **🎉 MILESTONE: Complete admin election with consensus, discovery protocol, heartbeats, and SLURP activation!** |
|
| Election manager | ✅ Running | Admin election integrated with Backbeat; metrics exposed under `pkg/metrics`. |
|
||||||
| SLURP (context intelligence) | 🚧 Stubbed | `pkg/slurp/slurp.go` contains TODOs for resolver, temporal graphs, intelligence. Leader integration scaffolding exists but uses placeholder IDs/request forwarding. |
|
| SLURP (context intelligence) | 🚧 Stubbed | `pkg/slurp/slurp.go` contains TODOs for resolver, temporal graphs, intelligence. Leader integration scaffolding exists but uses placeholder IDs/request forwarding. |
|
||||||
| SHHH (secrets sentinel) | 🚧 Sentinel live | `pkg/shhh` redacts hypercore + PubSub payloads with audit + metrics hooks (policy replay TBD). |
|
| SHHH (secrets sentinel) | 🚧 Sentinel live | `pkg/shhh` redacts hypercore + PubSub payloads with audit + metrics hooks (policy replay TBD). |
|
||||||
| HMMM routing | 🚧 Partial | PubSub topics join, but capability/role announcements and HMMM router wiring are placeholders (`internal/runtime/agent_support.go`). |
|
| HMMM routing | 🚧 Partial | PubSub topics join, but capability/role announcements and HMMM router wiring are placeholders (`internal/runtime/agent_support.go`). |
|
||||||
@@ -35,39 +35,6 @@ You’ll get a single agent container with:
|
|||||||
|
|
||||||
**Missing today:** SLURP context resolution, advanced SHHH policy replay, HMMM per-issue routing. Expect log warnings/TODOs for those paths.
|
**Missing today:** SLURP context resolution, advanced SHHH policy replay, HMMM per-issue routing. Expect log warnings/TODOs for those paths.
|
||||||
|
|
||||||
## 🎉 Leader Election System (NEW!)
|
|
||||||
|
|
||||||
CHORUS now features a complete, production-ready leader election system:
|
|
||||||
|
|
||||||
### Core Features
|
|
||||||
- **Consensus-based election** with weighted scoring (uptime, capabilities, resources)
|
|
||||||
- **Admin discovery protocol** for network-wide leader identification
|
|
||||||
- **Heartbeat system** with automatic failover (15-second intervals)
|
|
||||||
- **Concurrent election prevention** with randomized delays
|
|
||||||
- **SLURP activation** on elected admin nodes
|
|
||||||
|
|
||||||
### How It Works
|
|
||||||
1. **Bootstrap**: Nodes start in idle state, no admin known
|
|
||||||
2. **Discovery**: Nodes send discovery requests to find existing admin
|
|
||||||
3. **Election trigger**: If no admin found after grace period, trigger election
|
|
||||||
4. **Candidacy**: Eligible nodes announce themselves with capability scores
|
|
||||||
5. **Consensus**: Network selects winner based on highest score
|
|
||||||
6. **Leadership**: Winner starts heartbeats, activates SLURP functionality
|
|
||||||
7. **Monitoring**: Nodes continuously verify admin health via heartbeats
|
|
||||||
|
|
||||||
### Debugging
|
|
||||||
Use these log patterns to monitor election health:
|
|
||||||
```bash
|
|
||||||
# Monitor WHOAMI messages and leader identification
|
|
||||||
docker service logs CHORUS_chorus | grep "🤖 WHOAMI\|👑\|📡.*Discovered"
|
|
||||||
|
|
||||||
# Track election cycles
|
|
||||||
docker service logs CHORUS_chorus | grep "🗳️\|📢.*candidacy\|🏆.*winner"
|
|
||||||
|
|
||||||
# Watch discovery protocol
|
|
||||||
docker service logs CHORUS_chorus | grep "📩\|📤\|📥"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Roadmap Highlights
|
## Roadmap Highlights
|
||||||
|
|
||||||
1. **Security substrate** – land SHHH sentinel, finish SLURP leader-only operations, validate COOEE enrolment (see roadmap Phase 1).
|
1. **Security substrate** – land SHHH sentinel, finish SLURP leader-only operations, validate COOEE enrolment (see roadmap Phase 1).
|
||||||
|
|||||||
@@ -1,38 +0,0 @@
|
|||||||
{
|
|
||||||
"metadata": {
|
|
||||||
"generated_at": "2024-12-19T10:00:00Z",
|
|
||||||
"cluster_id": "production-cluster",
|
|
||||||
"version": "1.0.0",
|
|
||||||
"notes": "Bootstrap configuration for CHORUS scaling - managed by WHOOSH"
|
|
||||||
},
|
|
||||||
"peers": [
|
|
||||||
{
|
|
||||||
"address": "/ip4/10.0.1.10/tcp/9000/p2p/12D3KooWExample1234567890abcdef",
|
|
||||||
"priority": 100,
|
|
||||||
"region": "us-east-1",
|
|
||||||
"roles": ["admin", "stable"],
|
|
||||||
"enabled": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"address": "/ip4/10.0.1.11/tcp/9000/p2p/12D3KooWExample1234567890abcde2",
|
|
||||||
"priority": 90,
|
|
||||||
"region": "us-east-1",
|
|
||||||
"roles": ["worker", "stable"],
|
|
||||||
"enabled": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"address": "/ip4/10.0.2.10/tcp/9000/p2p/12D3KooWExample1234567890abcde3",
|
|
||||||
"priority": 80,
|
|
||||||
"region": "us-west-2",
|
|
||||||
"roles": ["worker", "stable"],
|
|
||||||
"enabled": true
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"address": "/ip4/10.0.3.10/tcp/9000/p2p/12D3KooWExample1234567890abcde4",
|
|
||||||
"priority": 70,
|
|
||||||
"region": "eu-central-1",
|
|
||||||
"roles": ["worker"],
|
|
||||||
"enabled": false
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
@@ -15,32 +15,13 @@ services:
|
|||||||
- CHORUS_AGENT_ID=${CHORUS_AGENT_ID:-} # Auto-generated if not provided
|
- CHORUS_AGENT_ID=${CHORUS_AGENT_ID:-} # Auto-generated if not provided
|
||||||
- CHORUS_SPECIALIZATION=${CHORUS_SPECIALIZATION:-general_developer}
|
- CHORUS_SPECIALIZATION=${CHORUS_SPECIALIZATION:-general_developer}
|
||||||
- CHORUS_MAX_TASKS=${CHORUS_MAX_TASKS:-3}
|
- CHORUS_MAX_TASKS=${CHORUS_MAX_TASKS:-3}
|
||||||
- CHORUS_CAPABILITIES=general_development,task_coordination,admin_election
|
- CHORUS_CAPABILITIES=${CHORUS_CAPABILITIES:-general_development,task_coordination,admin_election}
|
||||||
|
|
||||||
# Network configuration
|
# Network configuration
|
||||||
- CHORUS_API_PORT=8080
|
- CHORUS_API_PORT=8080
|
||||||
- CHORUS_HEALTH_PORT=8081
|
- CHORUS_HEALTH_PORT=8081
|
||||||
- CHORUS_P2P_PORT=9000
|
- CHORUS_P2P_PORT=9000
|
||||||
- CHORUS_BIND_ADDRESS=0.0.0.0
|
- CHORUS_BIND_ADDRESS=0.0.0.0
|
||||||
|
|
||||||
# Scaling optimizations (as per WHOOSH issue #7)
|
|
||||||
- CHORUS_MDNS_ENABLED=false # Disabled for container/swarm environments
|
|
||||||
- CHORUS_DIALS_PER_SEC=5 # Rate limit outbound connections to prevent storms
|
|
||||||
- CHORUS_MAX_CONCURRENT_DHT=16 # Limit concurrent DHT queries
|
|
||||||
|
|
||||||
# Election stability windows (Medium-risk fix 2.1)
|
|
||||||
- CHORUS_ELECTION_MIN_TERM=30s # Minimum time between elections to prevent churn
|
|
||||||
- CHORUS_LEADER_MIN_TERM=45s # Minimum time before challenging healthy leader
|
|
||||||
|
|
||||||
# Assignment system for runtime configuration (Medium-risk fix 2.2)
|
|
||||||
- ASSIGN_URL=${ASSIGN_URL:-} # Optional: WHOOSH assignment endpoint
|
|
||||||
- TASK_SLOT=${TASK_SLOT:-} # Optional: Task slot identifier
|
|
||||||
- TASK_ID=${TASK_ID:-} # Optional: Task identifier
|
|
||||||
- NODE_ID=${NODE_ID:-} # Optional: Node identifier
|
|
||||||
|
|
||||||
# Bootstrap pool configuration (supports JSON and CSV)
|
|
||||||
- BOOTSTRAP_JSON=/config/bootstrap.json # Optional: JSON bootstrap config
|
|
||||||
- CHORUS_BOOTSTRAP_PEERS=${CHORUS_BOOTSTRAP_PEERS:-} # CSV fallback
|
|
||||||
|
|
||||||
# AI configuration - Provider selection
|
# AI configuration - Provider selection
|
||||||
- CHORUS_AI_PROVIDER=${CHORUS_AI_PROVIDER:-resetdata}
|
- CHORUS_AI_PROVIDER=${CHORUS_AI_PROVIDER:-resetdata}
|
||||||
@@ -76,11 +57,6 @@ services:
|
|||||||
secrets:
|
secrets:
|
||||||
- chorus_license_id
|
- chorus_license_id
|
||||||
- resetdata_api_key
|
- resetdata_api_key
|
||||||
|
|
||||||
# Configuration files
|
|
||||||
configs:
|
|
||||||
- source: chorus_bootstrap
|
|
||||||
target: /config/bootstrap.json
|
|
||||||
|
|
||||||
# Persistent data storage
|
# Persistent data storage
|
||||||
volumes:
|
volumes:
|
||||||
@@ -193,7 +169,7 @@ services:
|
|||||||
# Scaling system configuration
|
# Scaling system configuration
|
||||||
WHOOSH_SCALING_KACHING_URL: "https://kaching.chorus.services"
|
WHOOSH_SCALING_KACHING_URL: "https://kaching.chorus.services"
|
||||||
WHOOSH_SCALING_BACKBEAT_URL: "http://backbeat-pulse:8080"
|
WHOOSH_SCALING_BACKBEAT_URL: "http://backbeat-pulse:8080"
|
||||||
WHOOSH_SCALING_CHORUS_URL: "http://chorus:9000"
|
WHOOSH_SCALING_CHORUS_URL: "http://chorus:8080"
|
||||||
secrets:
|
secrets:
|
||||||
- whoosh_db_password
|
- whoosh_db_password
|
||||||
- gitea_token
|
- gitea_token
|
||||||
@@ -640,10 +616,6 @@ networks:
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
configs:
|
|
||||||
chorus_bootstrap:
|
|
||||||
file: ./bootstrap.json
|
|
||||||
|
|
||||||
secrets:
|
secrets:
|
||||||
chorus_license_id:
|
chorus_license_id:
|
||||||
external: true
|
external: true
|
||||||
|
|||||||
@@ -105,7 +105,6 @@ func (t *SimpleTaskTracker) publishTaskCompletion(taskID string, success bool, s
|
|||||||
// SharedRuntime contains all the shared P2P infrastructure components
|
// SharedRuntime contains all the shared P2P infrastructure components
|
||||||
type SharedRuntime struct {
|
type SharedRuntime struct {
|
||||||
Config *config.Config
|
Config *config.Config
|
||||||
RuntimeConfig *config.RuntimeConfig
|
|
||||||
Logger *SimpleLogger
|
Logger *SimpleLogger
|
||||||
Context context.Context
|
Context context.Context
|
||||||
Cancel context.CancelFunc
|
Cancel context.CancelFunc
|
||||||
@@ -150,28 +149,6 @@ func Initialize(appMode string) (*SharedRuntime, error) {
|
|||||||
runtime.Config = cfg
|
runtime.Config = cfg
|
||||||
|
|
||||||
runtime.Logger.Info("✅ Configuration loaded successfully")
|
runtime.Logger.Info("✅ Configuration loaded successfully")
|
||||||
|
|
||||||
// Initialize runtime configuration with assignment support
|
|
||||||
runtime.RuntimeConfig = config.NewRuntimeConfig(cfg)
|
|
||||||
|
|
||||||
// Load assignment if ASSIGN_URL is configured
|
|
||||||
if assignURL := os.Getenv("ASSIGN_URL"); assignURL != "" {
|
|
||||||
runtime.Logger.Info("📡 Loading assignment from WHOOSH: %s", assignURL)
|
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(runtime.Context, 10*time.Second)
|
|
||||||
if err := runtime.RuntimeConfig.LoadAssignment(ctx, assignURL); err != nil {
|
|
||||||
runtime.Logger.Warn("⚠️ Failed to load assignment (continuing with base config): %v", err)
|
|
||||||
} else {
|
|
||||||
runtime.Logger.Info("✅ Assignment loaded successfully")
|
|
||||||
}
|
|
||||||
cancel()
|
|
||||||
|
|
||||||
// Start reload handler for SIGHUP
|
|
||||||
runtime.RuntimeConfig.StartReloadHandler(runtime.Context, assignURL)
|
|
||||||
runtime.Logger.Info("📡 SIGHUP reload handler started for assignment updates")
|
|
||||||
} else {
|
|
||||||
runtime.Logger.Info("⚪ No ASSIGN_URL configured, using static configuration")
|
|
||||||
}
|
|
||||||
runtime.Logger.Info("🤖 Agent ID: %s", cfg.Agent.ID)
|
runtime.Logger.Info("🤖 Agent ID: %s", cfg.Agent.ID)
|
||||||
runtime.Logger.Info("🎯 Specialization: %s", cfg.Agent.Specialization)
|
runtime.Logger.Info("🎯 Specialization: %s", cfg.Agent.Specialization)
|
||||||
|
|
||||||
@@ -248,17 +225,12 @@ func Initialize(appMode string) (*SharedRuntime, error) {
|
|||||||
runtime.HypercoreLog = hlog
|
runtime.HypercoreLog = hlog
|
||||||
runtime.Logger.Info("📝 Hypercore logger initialized")
|
runtime.Logger.Info("📝 Hypercore logger initialized")
|
||||||
|
|
||||||
// Initialize mDNS discovery (disabled in container environments for scaling)
|
// Initialize mDNS discovery
|
||||||
if cfg.V2.DHT.MDNSEnabled {
|
mdnsDiscovery, err := discovery.NewMDNSDiscovery(ctx, node.Host(), "chorus-peer-discovery")
|
||||||
mdnsDiscovery, err := discovery.NewMDNSDiscovery(ctx, node.Host(), "chorus-peer-discovery")
|
if err != nil {
|
||||||
if err != nil {
|
return nil, fmt.Errorf("failed to create mDNS discovery: %v", err)
|
||||||
return nil, fmt.Errorf("failed to create mDNS discovery: %v", err)
|
|
||||||
}
|
|
||||||
runtime.MDNSDiscovery = mdnsDiscovery
|
|
||||||
runtime.Logger.Info("🔍 mDNS discovery enabled for local network")
|
|
||||||
} else {
|
|
||||||
runtime.Logger.Info("⚪ mDNS discovery disabled (recommended for container/swarm deployments)")
|
|
||||||
}
|
}
|
||||||
|
runtime.MDNSDiscovery = mdnsDiscovery
|
||||||
|
|
||||||
// Initialize PubSub with hypercore logging
|
// Initialize PubSub with hypercore logging
|
||||||
ps, err := pubsub.NewPubSubWithLogger(ctx, node.Host(), "chorus/coordination/v1", "hmmm/meta-discussion/v1", hlog)
|
ps, err := pubsub.NewPubSubWithLogger(ctx, node.Host(), "chorus/coordination/v1", "hmmm/meta-discussion/v1", hlog)
|
||||||
@@ -311,7 +283,6 @@ func (r *SharedRuntime) Cleanup() {
|
|||||||
|
|
||||||
if r.MDNSDiscovery != nil {
|
if r.MDNSDiscovery != nil {
|
||||||
r.MDNSDiscovery.Close()
|
r.MDNSDiscovery.Close()
|
||||||
r.Logger.Info("🔍 mDNS discovery closed")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if r.PubSub != nil {
|
if r.PubSub != nil {
|
||||||
@@ -436,20 +407,8 @@ func (r *SharedRuntime) initializeDHTStorage() error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Connect to bootstrap peers (with assignment override support)
|
// Connect to bootstrap peers if configured
|
||||||
bootstrapPeers := r.RuntimeConfig.GetBootstrapPeers()
|
for _, addrStr := range r.Config.V2.DHT.BootstrapPeers {
|
||||||
if len(bootstrapPeers) == 0 {
|
|
||||||
bootstrapPeers = r.Config.V2.DHT.BootstrapPeers
|
|
||||||
}
|
|
||||||
|
|
||||||
// Apply join stagger if configured
|
|
||||||
joinStagger := r.RuntimeConfig.GetJoinStagger()
|
|
||||||
if joinStagger > 0 {
|
|
||||||
r.Logger.Info("⏱️ Applying join stagger delay: %v", joinStagger)
|
|
||||||
time.Sleep(joinStagger)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, addrStr := range bootstrapPeers {
|
|
||||||
addr, err := multiaddr.NewMultiaddr(addrStr)
|
addr, err := multiaddr.NewMultiaddr(addrStr)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
r.Logger.Warn("⚠️ Invalid bootstrap address %s: %v", addrStr, err)
|
r.Logger.Warn("⚠️ Invalid bootstrap address %s: %v", addrStr, err)
|
||||||
|
|||||||
16
p2p/node.go
16
p2p/node.go
@@ -9,7 +9,6 @@ import (
|
|||||||
"github.com/libp2p/go-libp2p"
|
"github.com/libp2p/go-libp2p"
|
||||||
"github.com/libp2p/go-libp2p/core/host"
|
"github.com/libp2p/go-libp2p/core/host"
|
||||||
"github.com/libp2p/go-libp2p/core/peer"
|
"github.com/libp2p/go-libp2p/core/peer"
|
||||||
"github.com/libp2p/go-libp2p/p2p/net/connmgr"
|
|
||||||
"github.com/libp2p/go-libp2p/p2p/security/noise"
|
"github.com/libp2p/go-libp2p/p2p/security/noise"
|
||||||
"github.com/libp2p/go-libp2p/p2p/transport/tcp"
|
"github.com/libp2p/go-libp2p/p2p/transport/tcp"
|
||||||
kaddht "github.com/libp2p/go-libp2p-kad-dht"
|
kaddht "github.com/libp2p/go-libp2p-kad-dht"
|
||||||
@@ -45,26 +44,13 @@ func NewNode(ctx context.Context, opts ...Option) (*Node, error) {
|
|||||||
listenAddrs = append(listenAddrs, ma)
|
listenAddrs = append(listenAddrs, ma)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create connection manager with scaling-optimized limits
|
// Create libp2p host with security and transport options
|
||||||
connManager, err := connmgr.NewConnManager(
|
|
||||||
config.LowWatermark, // Low watermark (32)
|
|
||||||
config.HighWatermark, // High watermark (128)
|
|
||||||
connmgr.WithGracePeriod(30*time.Second), // Grace period before pruning
|
|
||||||
)
|
|
||||||
if err != nil {
|
|
||||||
cancel()
|
|
||||||
return nil, fmt.Errorf("failed to create connection manager: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Create libp2p host with security, transport, and scaling options
|
|
||||||
h, err := libp2p.New(
|
h, err := libp2p.New(
|
||||||
libp2p.ListenAddrs(listenAddrs...),
|
libp2p.ListenAddrs(listenAddrs...),
|
||||||
libp2p.Security(noise.ID, noise.New),
|
libp2p.Security(noise.ID, noise.New),
|
||||||
libp2p.Transport(tcp.NewTCPTransport),
|
libp2p.Transport(tcp.NewTCPTransport),
|
||||||
libp2p.DefaultMuxers,
|
libp2p.DefaultMuxers,
|
||||||
libp2p.EnableRelay(),
|
libp2p.EnableRelay(),
|
||||||
libp2p.ConnectionManager(connManager), // Add connection management
|
|
||||||
libp2p.EnableAutoNATv2(), // Enable AutoNAT for container environments
|
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
cancel()
|
cancel()
|
||||||
|
|||||||
@@ -1,517 +0,0 @@
|
|||||||
package config
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"net/http"
|
|
||||||
"os"
|
|
||||||
"signal"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
|
||||||
"syscall"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// RuntimeConfig manages runtime configuration with assignment overrides
|
|
||||||
type RuntimeConfig struct {
|
|
||||||
Base *Config `json:"base"`
|
|
||||||
Override *AssignmentConfig `json:"override"`
|
|
||||||
mu sync.RWMutex
|
|
||||||
reloadCh chan struct{}
|
|
||||||
}
|
|
||||||
|
|
||||||
// AssignmentConfig represents runtime assignment from WHOOSH
|
|
||||||
type AssignmentConfig struct {
|
|
||||||
// Assignment metadata
|
|
||||||
AssignmentID string `json:"assignment_id"`
|
|
||||||
TaskSlot string `json:"task_slot"`
|
|
||||||
TaskID string `json:"task_id"`
|
|
||||||
ClusterID string `json:"cluster_id"`
|
|
||||||
AssignedAt time.Time `json:"assigned_at"`
|
|
||||||
ExpiresAt time.Time `json:"expires_at,omitempty"`
|
|
||||||
|
|
||||||
// Agent configuration overrides
|
|
||||||
Agent *AgentConfig `json:"agent,omitempty"`
|
|
||||||
Network *NetworkConfig `json:"network,omitempty"`
|
|
||||||
AI *AIConfig `json:"ai,omitempty"`
|
|
||||||
Logging *LoggingConfig `json:"logging,omitempty"`
|
|
||||||
|
|
||||||
// Bootstrap configuration for scaling
|
|
||||||
BootstrapPeers []string `json:"bootstrap_peers,omitempty"`
|
|
||||||
JoinStagger int `json:"join_stagger_ms,omitempty"`
|
|
||||||
|
|
||||||
// Runtime capabilities
|
|
||||||
RuntimeCapabilities []string `json:"runtime_capabilities,omitempty"`
|
|
||||||
|
|
||||||
// Key derivation for encryption
|
|
||||||
RoleKey string `json:"role_key,omitempty"`
|
|
||||||
ClusterSecret string `json:"cluster_secret,omitempty"`
|
|
||||||
|
|
||||||
// Custom fields
|
|
||||||
Custom map[string]interface{} `json:"custom,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// AssignmentRequest represents a request for assignment from WHOOSH
|
|
||||||
type AssignmentRequest struct {
|
|
||||||
ClusterID string `json:"cluster_id"`
|
|
||||||
TaskSlot string `json:"task_slot,omitempty"`
|
|
||||||
TaskID string `json:"task_id,omitempty"`
|
|
||||||
AgentID string `json:"agent_id"`
|
|
||||||
NodeID string `json:"node_id"`
|
|
||||||
Timestamp time.Time `json:"timestamp"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// NewRuntimeConfig creates a new runtime configuration manager
|
|
||||||
func NewRuntimeConfig(baseConfig *Config) *RuntimeConfig {
|
|
||||||
return &RuntimeConfig{
|
|
||||||
Base: baseConfig,
|
|
||||||
Override: nil,
|
|
||||||
reloadCh: make(chan struct{}, 1),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get returns the effective configuration value, with override taking precedence
|
|
||||||
func (rc *RuntimeConfig) Get(field string) interface{} {
|
|
||||||
rc.mu.RLock()
|
|
||||||
defer rc.mu.RUnlock()
|
|
||||||
|
|
||||||
// Try override first
|
|
||||||
if rc.Override != nil {
|
|
||||||
if value := rc.getFromAssignment(field); value != nil {
|
|
||||||
return value
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Fall back to base configuration
|
|
||||||
return rc.getFromBase(field)
|
|
||||||
}
|
|
||||||
|
|
||||||
// GetConfig returns a merged configuration with overrides applied
|
|
||||||
func (rc *RuntimeConfig) GetConfig() *Config {
|
|
||||||
rc.mu.RLock()
|
|
||||||
defer rc.mu.RUnlock()
|
|
||||||
|
|
||||||
if rc.Override == nil {
|
|
||||||
return rc.Base
|
|
||||||
}
|
|
||||||
|
|
||||||
// Create a copy of base config
|
|
||||||
merged := *rc.Base
|
|
||||||
|
|
||||||
// Apply overrides
|
|
||||||
if rc.Override.Agent != nil {
|
|
||||||
rc.mergeAgentConfig(&merged.Agent, rc.Override.Agent)
|
|
||||||
}
|
|
||||||
if rc.Override.Network != nil {
|
|
||||||
rc.mergeNetworkConfig(&merged.Network, rc.Override.Network)
|
|
||||||
}
|
|
||||||
if rc.Override.AI != nil {
|
|
||||||
rc.mergeAIConfig(&merged.AI, rc.Override.AI)
|
|
||||||
}
|
|
||||||
if rc.Override.Logging != nil {
|
|
||||||
rc.mergeLoggingConfig(&merged.Logging, rc.Override.Logging)
|
|
||||||
}
|
|
||||||
|
|
||||||
return &merged
|
|
||||||
}
|
|
||||||
|
|
||||||
// LoadAssignment fetches assignment from WHOOSH and applies it
|
|
||||||
func (rc *RuntimeConfig) LoadAssignment(ctx context.Context, assignURL string) error {
|
|
||||||
if assignURL == "" {
|
|
||||||
return nil // No assignment URL configured
|
|
||||||
}
|
|
||||||
|
|
||||||
// Build assignment request
|
|
||||||
agentID := rc.Base.Agent.ID
|
|
||||||
if agentID == "" {
|
|
||||||
agentID = "unknown"
|
|
||||||
}
|
|
||||||
|
|
||||||
req := AssignmentRequest{
|
|
||||||
ClusterID: rc.Base.License.ClusterID,
|
|
||||||
TaskSlot: os.Getenv("TASK_SLOT"),
|
|
||||||
TaskID: os.Getenv("TASK_ID"),
|
|
||||||
AgentID: agentID,
|
|
||||||
NodeID: os.Getenv("NODE_ID"),
|
|
||||||
Timestamp: time.Now(),
|
|
||||||
}
|
|
||||||
|
|
||||||
// Make HTTP request to WHOOSH
|
|
||||||
assignment, err := rc.fetchAssignment(ctx, assignURL, req)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to fetch assignment: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Apply assignment
|
|
||||||
rc.mu.Lock()
|
|
||||||
rc.Override = assignment
|
|
||||||
rc.mu.Unlock()
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// StartReloadHandler starts a signal handler for SIGHUP configuration reloads
|
|
||||||
func (rc *RuntimeConfig) StartReloadHandler(ctx context.Context, assignURL string) {
|
|
||||||
sigCh := make(chan os.Signal, 1)
|
|
||||||
signal.Notify(sigCh, syscall.SIGHUP)
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case <-sigCh:
|
|
||||||
fmt.Println("📡 Received SIGHUP, reloading assignment configuration...")
|
|
||||||
if err := rc.LoadAssignment(ctx, assignURL); err != nil {
|
|
||||||
fmt.Printf("❌ Failed to reload assignment: %v\n", err)
|
|
||||||
} else {
|
|
||||||
fmt.Println("✅ Assignment configuration reloaded successfully")
|
|
||||||
}
|
|
||||||
case <-rc.reloadCh:
|
|
||||||
// Manual reload trigger
|
|
||||||
if err := rc.LoadAssignment(ctx, assignURL); err != nil {
|
|
||||||
fmt.Printf("❌ Failed to reload assignment: %v\n", err)
|
|
||||||
} else {
|
|
||||||
fmt.Println("✅ Assignment configuration reloaded successfully")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Reload triggers a manual configuration reload
|
|
||||||
func (rc *RuntimeConfig) Reload() {
|
|
||||||
select {
|
|
||||||
case rc.reloadCh <- struct{}{}:
|
|
||||||
default:
|
|
||||||
// Channel full, reload already pending
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// fetchAssignment makes HTTP request to WHOOSH assignment API
|
|
||||||
func (rc *RuntimeConfig) fetchAssignment(ctx context.Context, assignURL string, req AssignmentRequest) (*AssignmentConfig, error) {
|
|
||||||
// Build query parameters
|
|
||||||
queryParams := fmt.Sprintf("?cluster_id=%s&agent_id=%s&node_id=%s",
|
|
||||||
req.ClusterID, req.AgentID, req.NodeID)
|
|
||||||
|
|
||||||
if req.TaskSlot != "" {
|
|
||||||
queryParams += "&task_slot=" + req.TaskSlot
|
|
||||||
}
|
|
||||||
if req.TaskID != "" {
|
|
||||||
queryParams += "&task_id=" + req.TaskID
|
|
||||||
}
|
|
||||||
|
|
||||||
// Create HTTP request
|
|
||||||
httpReq, err := http.NewRequestWithContext(ctx, "GET", assignURL+queryParams, nil)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to create assignment request: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
httpReq.Header.Set("Accept", "application/json")
|
|
||||||
httpReq.Header.Set("User-Agent", "CHORUS-Agent/0.1.0")
|
|
||||||
|
|
||||||
// Make request with timeout
|
|
||||||
client := &http.Client{Timeout: 10 * time.Second}
|
|
||||||
resp, err := client.Do(httpReq)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("assignment request failed: %w", err)
|
|
||||||
}
|
|
||||||
defer resp.Body.Close()
|
|
||||||
|
|
||||||
if resp.StatusCode == http.StatusNotFound {
|
|
||||||
// No assignment available
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if resp.StatusCode != http.StatusOK {
|
|
||||||
body, _ := io.ReadAll(resp.Body)
|
|
||||||
return nil, fmt.Errorf("assignment request failed with status %d: %s", resp.StatusCode, string(body))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Parse assignment response
|
|
||||||
var assignment AssignmentConfig
|
|
||||||
if err := json.NewDecoder(resp.Body).Decode(&assignment); err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to decode assignment response: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return &assignment, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Helper methods for getting values from different sources
|
|
||||||
func (rc *RuntimeConfig) getFromAssignment(field string) interface{} {
|
|
||||||
if rc.Override == nil {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Simple field mapping - in a real implementation, you'd use reflection
|
|
||||||
// or a more sophisticated field mapping system
|
|
||||||
switch field {
|
|
||||||
case "agent.id":
|
|
||||||
if rc.Override.Agent != nil && rc.Override.Agent.ID != "" {
|
|
||||||
return rc.Override.Agent.ID
|
|
||||||
}
|
|
||||||
case "agent.role":
|
|
||||||
if rc.Override.Agent != nil && rc.Override.Agent.Role != "" {
|
|
||||||
return rc.Override.Agent.Role
|
|
||||||
}
|
|
||||||
case "agent.capabilities":
|
|
||||||
if len(rc.Override.RuntimeCapabilities) > 0 {
|
|
||||||
return rc.Override.RuntimeCapabilities
|
|
||||||
}
|
|
||||||
case "bootstrap_peers":
|
|
||||||
if len(rc.Override.BootstrapPeers) > 0 {
|
|
||||||
return rc.Override.BootstrapPeers
|
|
||||||
}
|
|
||||||
case "join_stagger":
|
|
||||||
if rc.Override.JoinStagger > 0 {
|
|
||||||
return rc.Override.JoinStagger
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check custom fields
|
|
||||||
if rc.Override.Custom != nil {
|
|
||||||
if val, exists := rc.Override.Custom[field]; exists {
|
|
||||||
return val
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (rc *RuntimeConfig) getFromBase(field string) interface{} {
|
|
||||||
// Simple field mapping for base config
|
|
||||||
switch field {
|
|
||||||
case "agent.id":
|
|
||||||
return rc.Base.Agent.ID
|
|
||||||
case "agent.role":
|
|
||||||
return rc.Base.Agent.Role
|
|
||||||
case "agent.capabilities":
|
|
||||||
return rc.Base.Agent.Capabilities
|
|
||||||
default:
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Helper methods for merging configuration sections
|
|
||||||
func (rc *RuntimeConfig) mergeAgentConfig(base *AgentConfig, override *AgentConfig) {
|
|
||||||
if override.ID != "" {
|
|
||||||
base.ID = override.ID
|
|
||||||
}
|
|
||||||
if override.Specialization != "" {
|
|
||||||
base.Specialization = override.Specialization
|
|
||||||
}
|
|
||||||
if override.MaxTasks > 0 {
|
|
||||||
base.MaxTasks = override.MaxTasks
|
|
||||||
}
|
|
||||||
if len(override.Capabilities) > 0 {
|
|
||||||
base.Capabilities = override.Capabilities
|
|
||||||
}
|
|
||||||
if len(override.Models) > 0 {
|
|
||||||
base.Models = override.Models
|
|
||||||
}
|
|
||||||
if override.Role != "" {
|
|
||||||
base.Role = override.Role
|
|
||||||
}
|
|
||||||
if override.Project != "" {
|
|
||||||
base.Project = override.Project
|
|
||||||
}
|
|
||||||
if len(override.Expertise) > 0 {
|
|
||||||
base.Expertise = override.Expertise
|
|
||||||
}
|
|
||||||
if override.ReportsTo != "" {
|
|
||||||
base.ReportsTo = override.ReportsTo
|
|
||||||
}
|
|
||||||
if len(override.Deliverables) > 0 {
|
|
||||||
base.Deliverables = override.Deliverables
|
|
||||||
}
|
|
||||||
if override.ModelSelectionWebhook != "" {
|
|
||||||
base.ModelSelectionWebhook = override.ModelSelectionWebhook
|
|
||||||
}
|
|
||||||
if override.DefaultReasoningModel != "" {
|
|
||||||
base.DefaultReasoningModel = override.DefaultReasoningModel
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (rc *RuntimeConfig) mergeNetworkConfig(base *NetworkConfig, override *NetworkConfig) {
|
|
||||||
if override.P2PPort > 0 {
|
|
||||||
base.P2PPort = override.P2PPort
|
|
||||||
}
|
|
||||||
if override.APIPort > 0 {
|
|
||||||
base.APIPort = override.APIPort
|
|
||||||
}
|
|
||||||
if override.HealthPort > 0 {
|
|
||||||
base.HealthPort = override.HealthPort
|
|
||||||
}
|
|
||||||
if override.BindAddr != "" {
|
|
||||||
base.BindAddr = override.BindAddr
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (rc *RuntimeConfig) mergeAIConfig(base *AIConfig, override *AIConfig) {
|
|
||||||
if override.Provider != "" {
|
|
||||||
base.Provider = override.Provider
|
|
||||||
}
|
|
||||||
// Merge Ollama config if present
|
|
||||||
if override.Ollama.Endpoint != "" {
|
|
||||||
base.Ollama.Endpoint = override.Ollama.Endpoint
|
|
||||||
}
|
|
||||||
if override.Ollama.Timeout > 0 {
|
|
||||||
base.Ollama.Timeout = override.Ollama.Timeout
|
|
||||||
}
|
|
||||||
// Merge ResetData config if present
|
|
||||||
if override.ResetData.BaseURL != "" {
|
|
||||||
base.ResetData.BaseURL = override.ResetData.BaseURL
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (rc *RuntimeConfig) mergeLoggingConfig(base *LoggingConfig, override *LoggingConfig) {
|
|
||||||
if override.Level != "" {
|
|
||||||
base.Level = override.Level
|
|
||||||
}
|
|
||||||
if override.Format != "" {
|
|
||||||
base.Format = override.Format
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// BootstrapConfig represents JSON bootstrap configuration
|
|
||||||
type BootstrapConfig struct {
|
|
||||||
Peers []BootstrapPeer `json:"peers"`
|
|
||||||
Metadata BootstrapMeta `json:"metadata,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// BootstrapPeer represents a single bootstrap peer
|
|
||||||
type BootstrapPeer struct {
|
|
||||||
Address string `json:"address"`
|
|
||||||
Priority int `json:"priority,omitempty"`
|
|
||||||
Region string `json:"region,omitempty"`
|
|
||||||
Roles []string `json:"roles,omitempty"`
|
|
||||||
Enabled bool `json:"enabled"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// BootstrapMeta contains metadata about the bootstrap configuration
|
|
||||||
type BootstrapMeta struct {
|
|
||||||
GeneratedAt time.Time `json:"generated_at,omitempty"`
|
|
||||||
ClusterID string `json:"cluster_id,omitempty"`
|
|
||||||
Version string `json:"version,omitempty"`
|
|
||||||
Notes string `json:"notes,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// GetBootstrapPeers returns bootstrap peers with assignment override support and JSON config
|
|
||||||
func (rc *RuntimeConfig) GetBootstrapPeers() []string {
|
|
||||||
rc.mu.RLock()
|
|
||||||
defer rc.mu.RUnlock()
|
|
||||||
|
|
||||||
// First priority: Assignment override from WHOOSH
|
|
||||||
if rc.Override != nil && len(rc.Override.BootstrapPeers) > 0 {
|
|
||||||
return rc.Override.BootstrapPeers
|
|
||||||
}
|
|
||||||
|
|
||||||
// Second priority: JSON bootstrap configuration
|
|
||||||
if jsonPeers := rc.loadBootstrapJSON(); len(jsonPeers) > 0 {
|
|
||||||
return jsonPeers
|
|
||||||
}
|
|
||||||
|
|
||||||
// Third priority: Environment variable (CSV format)
|
|
||||||
if bootstrapEnv := os.Getenv("CHORUS_BOOTSTRAP_PEERS"); bootstrapEnv != "" {
|
|
||||||
peers := strings.Split(bootstrapEnv, ",")
|
|
||||||
// Trim whitespace from each peer
|
|
||||||
for i, peer := range peers {
|
|
||||||
peers[i] = strings.TrimSpace(peer)
|
|
||||||
}
|
|
||||||
return peers
|
|
||||||
}
|
|
||||||
|
|
||||||
return []string{}
|
|
||||||
}
|
|
||||||
|
|
||||||
// loadBootstrapJSON loads bootstrap peers from JSON file
|
|
||||||
func (rc *RuntimeConfig) loadBootstrapJSON() []string {
|
|
||||||
jsonPath := os.Getenv("BOOTSTRAP_JSON")
|
|
||||||
if jsonPath == "" {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check if file exists
|
|
||||||
if _, err := os.Stat(jsonPath); os.IsNotExist(err) {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Read and parse JSON file
|
|
||||||
data, err := os.ReadFile(jsonPath)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Printf("⚠️ Failed to read bootstrap JSON file %s: %v\n", jsonPath, err)
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
var config BootstrapConfig
|
|
||||||
if err := json.Unmarshal(data, &config); err != nil {
|
|
||||||
fmt.Printf("⚠️ Failed to parse bootstrap JSON file %s: %v\n", jsonPath, err)
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Extract enabled peer addresses, sorted by priority
|
|
||||||
var peers []string
|
|
||||||
enabledPeers := make([]BootstrapPeer, 0, len(config.Peers))
|
|
||||||
|
|
||||||
// Filter enabled peers
|
|
||||||
for _, peer := range config.Peers {
|
|
||||||
if peer.Enabled && peer.Address != "" {
|
|
||||||
enabledPeers = append(enabledPeers, peer)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Sort by priority (higher priority first)
|
|
||||||
for i := 0; i < len(enabledPeers)-1; i++ {
|
|
||||||
for j := i + 1; j < len(enabledPeers); j++ {
|
|
||||||
if enabledPeers[j].Priority > enabledPeers[i].Priority {
|
|
||||||
enabledPeers[i], enabledPeers[j] = enabledPeers[j], enabledPeers[i]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Extract addresses
|
|
||||||
for _, peer := range enabledPeers {
|
|
||||||
peers = append(peers, peer.Address)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(peers) > 0 {
|
|
||||||
fmt.Printf("📋 Loaded %d bootstrap peers from JSON: %s\n", len(peers), jsonPath)
|
|
||||||
}
|
|
||||||
|
|
||||||
return peers
|
|
||||||
}
|
|
||||||
|
|
||||||
// GetJoinStagger returns join stagger delay with assignment override support
|
|
||||||
func (rc *RuntimeConfig) GetJoinStagger() time.Duration {
|
|
||||||
rc.mu.RLock()
|
|
||||||
defer rc.mu.RUnlock()
|
|
||||||
|
|
||||||
if rc.Override != nil && rc.Override.JoinStagger > 0 {
|
|
||||||
return time.Duration(rc.Override.JoinStagger) * time.Millisecond
|
|
||||||
}
|
|
||||||
|
|
||||||
// Fall back to environment variable
|
|
||||||
if staggerEnv := os.Getenv("CHORUS_JOIN_STAGGER_MS"); staggerEnv != "" {
|
|
||||||
if ms, err := time.ParseDuration(staggerEnv + "ms"); err == nil {
|
|
||||||
return ms
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// GetAssignmentInfo returns current assignment metadata
|
|
||||||
func (rc *RuntimeConfig) GetAssignmentInfo() *AssignmentConfig {
|
|
||||||
rc.mu.RLock()
|
|
||||||
defer rc.mu.RUnlock()
|
|
||||||
|
|
||||||
if rc.Override == nil {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// Return a copy to prevent external modification
|
|
||||||
assignment := *rc.Override
|
|
||||||
return &assignment
|
|
||||||
}
|
|
||||||
@@ -41,16 +41,10 @@ type HybridUCXLConfig struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type DiscoveryConfig struct {
|
type DiscoveryConfig struct {
|
||||||
MDNSEnabled bool `env:"CHORUS_MDNS_ENABLED" default:"true" json:"mdns_enabled" yaml:"mdns_enabled"`
|
MDNSEnabled bool `env:"CHORUS_MDNS_ENABLED" default:"true" json:"mdns_enabled" yaml:"mdns_enabled"`
|
||||||
DHTDiscovery bool `env:"CHORUS_DHT_DISCOVERY" default:"false" json:"dht_discovery" yaml:"dht_discovery"`
|
DHTDiscovery bool `env:"CHORUS_DHT_DISCOVERY" default:"false" json:"dht_discovery" yaml:"dht_discovery"`
|
||||||
AnnounceInterval time.Duration `env:"CHORUS_ANNOUNCE_INTERVAL" default:"30s" json:"announce_interval" yaml:"announce_interval"`
|
AnnounceInterval time.Duration `env:"CHORUS_ANNOUNCE_INTERVAL" default:"30s" json:"announce_interval" yaml:"announce_interval"`
|
||||||
ServiceName string `env:"CHORUS_SERVICE_NAME" default:"CHORUS" json:"service_name" yaml:"service_name"`
|
ServiceName string `env:"CHORUS_SERVICE_NAME" default:"CHORUS" json:"service_name" yaml:"service_name"`
|
||||||
|
|
||||||
// Rate limiting for scaling (as per WHOOSH issue #7)
|
|
||||||
DialsPerSecond int `env:"CHORUS_DIALS_PER_SEC" default:"5" json:"dials_per_second" yaml:"dials_per_second"`
|
|
||||||
MaxConcurrentDHT int `env:"CHORUS_MAX_CONCURRENT_DHT" default:"16" json:"max_concurrent_dht" yaml:"max_concurrent_dht"`
|
|
||||||
MaxConcurrentDials int `env:"CHORUS_MAX_CONCURRENT_DIALS" default:"10" json:"max_concurrent_dials" yaml:"max_concurrent_dials"`
|
|
||||||
JoinStaggerMS int `env:"CHORUS_JOIN_STAGGER_MS" default:"0" json:"join_stagger_ms" yaml:"join_stagger_ms"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type MonitoringConfig struct {
|
type MonitoringConfig struct {
|
||||||
@@ -85,16 +79,10 @@ func LoadHybridConfig() (*HybridConfig, error) {
|
|||||||
|
|
||||||
// Load Discovery configuration
|
// Load Discovery configuration
|
||||||
config.Discovery = DiscoveryConfig{
|
config.Discovery = DiscoveryConfig{
|
||||||
MDNSEnabled: getEnvBool("CHORUS_MDNS_ENABLED", true),
|
MDNSEnabled: getEnvBool("CHORUS_MDNS_ENABLED", true),
|
||||||
DHTDiscovery: getEnvBool("CHORUS_DHT_DISCOVERY", false),
|
DHTDiscovery: getEnvBool("CHORUS_DHT_DISCOVERY", false),
|
||||||
AnnounceInterval: getEnvDuration("CHORUS_ANNOUNCE_INTERVAL", 30*time.Second),
|
AnnounceInterval: getEnvDuration("CHORUS_ANNOUNCE_INTERVAL", 30*time.Second),
|
||||||
ServiceName: getEnvString("CHORUS_SERVICE_NAME", "CHORUS"),
|
ServiceName: getEnvString("CHORUS_SERVICE_NAME", "CHORUS"),
|
||||||
|
|
||||||
// Rate limiting for scaling (as per WHOOSH issue #7)
|
|
||||||
DialsPerSecond: getEnvInt("CHORUS_DIALS_PER_SEC", 5),
|
|
||||||
MaxConcurrentDHT: getEnvInt("CHORUS_MAX_CONCURRENT_DHT", 16),
|
|
||||||
MaxConcurrentDials: getEnvInt("CHORUS_MAX_CONCURRENT_DIALS", 10),
|
|
||||||
JoinStaggerMS: getEnvInt("CHORUS_JOIN_STAGGER_MS", 0),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Load Monitoring configuration
|
// Load Monitoring configuration
|
||||||
|
|||||||
@@ -6,7 +6,6 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"log"
|
"log"
|
||||||
"math/rand"
|
"math/rand"
|
||||||
"os"
|
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -103,11 +102,6 @@ type ElectionManager struct {
|
|||||||
onAdminChanged func(oldAdmin, newAdmin string)
|
onAdminChanged func(oldAdmin, newAdmin string)
|
||||||
onElectionComplete func(winner string)
|
onElectionComplete func(winner string)
|
||||||
|
|
||||||
// Stability window to prevent election churn (Medium-risk fix 2.1)
|
|
||||||
lastElectionTime time.Time
|
|
||||||
electionStabilityWindow time.Duration
|
|
||||||
leaderStabilityWindow time.Duration
|
|
||||||
|
|
||||||
startTime time.Time
|
startTime time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -143,10 +137,6 @@ func NewElectionManager(
|
|||||||
votes: make(map[string]string),
|
votes: make(map[string]string),
|
||||||
electionTrigger: make(chan ElectionTrigger, 10),
|
electionTrigger: make(chan ElectionTrigger, 10),
|
||||||
startTime: time.Now(),
|
startTime: time.Now(),
|
||||||
|
|
||||||
// Initialize stability windows (as per WHOOSH issue #7)
|
|
||||||
electionStabilityWindow: getElectionStabilityWindow(cfg),
|
|
||||||
leaderStabilityWindow: getLeaderStabilityWindow(cfg),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Initialize heartbeat manager
|
// Initialize heartbeat manager
|
||||||
@@ -230,13 +220,11 @@ func (em *ElectionManager) Stop() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TriggerElection manually triggers an election with stability window checks
|
// TriggerElection manually triggers an election
|
||||||
func (em *ElectionManager) TriggerElection(trigger ElectionTrigger) {
|
func (em *ElectionManager) TriggerElection(trigger ElectionTrigger) {
|
||||||
// Check if election already in progress
|
// Check if election already in progress
|
||||||
em.mu.RLock()
|
em.mu.RLock()
|
||||||
currentState := em.state
|
currentState := em.state
|
||||||
currentAdmin := em.currentAdmin
|
|
||||||
lastElection := em.lastElectionTime
|
|
||||||
em.mu.RUnlock()
|
em.mu.RUnlock()
|
||||||
|
|
||||||
if currentState != StateIdle {
|
if currentState != StateIdle {
|
||||||
@@ -244,26 +232,6 @@ func (em *ElectionManager) TriggerElection(trigger ElectionTrigger) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// Apply stability window to prevent election churn (WHOOSH issue #7)
|
|
||||||
now := time.Now()
|
|
||||||
if !lastElection.IsZero() {
|
|
||||||
timeSinceElection := now.Sub(lastElection)
|
|
||||||
|
|
||||||
// If we have a current admin, check leader stability window
|
|
||||||
if currentAdmin != "" && timeSinceElection < em.leaderStabilityWindow {
|
|
||||||
log.Printf("⏳ Leader stability window active (%.1fs remaining), ignoring trigger: %s",
|
|
||||||
(em.leaderStabilityWindow - timeSinceElection).Seconds(), trigger)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// General election stability window
|
|
||||||
if timeSinceElection < em.electionStabilityWindow {
|
|
||||||
log.Printf("⏳ Election stability window active (%.1fs remaining), ignoring trigger: %s",
|
|
||||||
(em.electionStabilityWindow - timeSinceElection).Seconds(), trigger)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
select {
|
select {
|
||||||
case em.electionTrigger <- trigger:
|
case em.electionTrigger <- trigger:
|
||||||
log.Printf("🗳️ Election triggered: %s", trigger)
|
log.Printf("🗳️ Election triggered: %s", trigger)
|
||||||
@@ -474,7 +442,6 @@ func (em *ElectionManager) beginElection(trigger ElectionTrigger) {
|
|||||||
em.mu.Lock()
|
em.mu.Lock()
|
||||||
em.state = StateElecting
|
em.state = StateElecting
|
||||||
em.currentTerm++
|
em.currentTerm++
|
||||||
em.lastElectionTime = time.Now() // Record election timestamp for stability window
|
|
||||||
term := em.currentTerm
|
term := em.currentTerm
|
||||||
em.candidates = make(map[string]*AdminCandidate)
|
em.candidates = make(map[string]*AdminCandidate)
|
||||||
em.votes = make(map[string]string)
|
em.votes = make(map[string]string)
|
||||||
@@ -1152,43 +1119,3 @@ func (hm *HeartbeatManager) GetHeartbeatStatus() map[string]interface{} {
|
|||||||
|
|
||||||
return status
|
return status
|
||||||
}
|
}
|
||||||
|
|
||||||
// Helper functions for stability window configuration
|
|
||||||
|
|
||||||
// getElectionStabilityWindow gets the minimum time between elections
|
|
||||||
func getElectionStabilityWindow(cfg *config.Config) time.Duration {
|
|
||||||
// Try to get from environment or use default
|
|
||||||
if stability := os.Getenv("CHORUS_ELECTION_MIN_TERM"); stability != "" {
|
|
||||||
if duration, err := time.ParseDuration(stability); err == nil {
|
|
||||||
return duration
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Try to get from config structure if it exists
|
|
||||||
if cfg.Security.ElectionConfig.DiscoveryTimeout > 0 {
|
|
||||||
// Use double the discovery timeout as default stability window
|
|
||||||
return cfg.Security.ElectionConfig.DiscoveryTimeout * 2
|
|
||||||
}
|
|
||||||
|
|
||||||
// Default fallback
|
|
||||||
return 30 * time.Second
|
|
||||||
}
|
|
||||||
|
|
||||||
// getLeaderStabilityWindow gets the minimum time before challenging a healthy leader
|
|
||||||
func getLeaderStabilityWindow(cfg *config.Config) time.Duration {
|
|
||||||
// Try to get from environment or use default
|
|
||||||
if stability := os.Getenv("CHORUS_LEADER_MIN_TERM"); stability != "" {
|
|
||||||
if duration, err := time.ParseDuration(stability); err == nil {
|
|
||||||
return duration
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Try to get from config structure if it exists
|
|
||||||
if cfg.Security.ElectionConfig.HeartbeatTimeout > 0 {
|
|
||||||
// Use 3x heartbeat timeout as default leader stability
|
|
||||||
return cfg.Security.ElectionConfig.HeartbeatTimeout * 3
|
|
||||||
}
|
|
||||||
|
|
||||||
// Default fallback
|
|
||||||
return 45 * time.Second
|
|
||||||
}
|
|
||||||
|
|||||||
Reference in New Issue
Block a user