Major milestone: CHORUS leader election is now fully functional! ## Key Features Implemented: ### 🗳️ Leader Election Core - Fixed root cause: nodes now trigger elections when no admin exists - Added randomized election delays to prevent simultaneous elections - Implemented concurrent election prevention (only one election at a time) - Added proper election state management and transitions ### 📡 Admin Discovery System - Enhanced discovery requests with "WHOAMI" debug messages - Fixed discovery responses to properly include current leader ID - Added comprehensive discovery request/response logging - Implemented admin confirmation from multiple sources ### 🔧 Configuration Improvements - Increased discovery timeout from 3s to 15s for better reliability - Added proper Docker Hub image deployment workflow - Updated build process to use correct chorus-agent binary (not deprecated chorus) - Added static compilation flags for Alpine Linux compatibility ### 🐛 Critical Fixes - Fixed build process confusion between chorus vs chorus-agent binaries - Added missing admin_election capability to enable leader elections - Corrected discovery logic to handle zero admin responses - Enhanced debugging with detailed state and timing information ## Current Operational Status: ✅ Admin Election: Working with proper consensus ✅ Heartbeat System: 15-second intervals from elected admin ✅ Discovery Protocol: Nodes can find and confirm current admin ✅ P2P Connectivity: 5+ connected peers with libp2p ✅ SLURP Functionality: Enabled on admin nodes ✅ BACKBEAT Integration: Tempo synchronization working ✅ Container Health: All health checks passing ## Technical Details: - Election uses weighted scoring based on uptime, capabilities, and resources - Randomized delays prevent election storms (30-45s wait periods) - Discovery responses include current leader ID for network-wide consensus - State management prevents multiple concurrent elections - Enhanced logging provides full visibility into election process 🎉 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
3.8 KiB
gobreaker
gobreaker implements the Circuit Breaker pattern in Go.
Installation
go get github.com/sony/gobreaker
Usage
The struct CircuitBreaker is a state machine to prevent sending requests that are likely to fail.
The function NewCircuitBreaker creates a new CircuitBreaker.
func NewCircuitBreaker(st Settings) *CircuitBreaker
You can configure CircuitBreaker by the struct Settings:
type Settings struct {
Name string
MaxRequests uint32
Interval time.Duration
Timeout time.Duration
ReadyToTrip func(counts Counts) bool
OnStateChange func(name string, from State, to State)
IsSuccessful func(err error) bool
}
-
Nameis the name of theCircuitBreaker. -
MaxRequestsis the maximum number of requests allowed to pass through when theCircuitBreakeris half-open. IfMaxRequestsis 0,CircuitBreakerallows only 1 request. -
Intervalis the cyclic period of the closed state forCircuitBreakerto clear the internalCounts, described later in this section. IfIntervalis 0,CircuitBreakerdoesn't clear the internalCountsduring the closed state. -
Timeoutis the period of the open state, after which the state ofCircuitBreakerbecomes half-open. IfTimeoutis 0, the timeout value ofCircuitBreakeris set to 60 seconds. -
ReadyToTripis called with a copy ofCountswhenever a request fails in the closed state. IfReadyToTripreturns true,CircuitBreakerwill be placed into the open state. IfReadyToTripisnil, defaultReadyToTripis used. DefaultReadyToTripreturns true when the number of consecutive failures is more than 5. -
OnStateChangeis called whenever the state ofCircuitBreakerchanges. -
IsSuccessfulis called with the error returned from a request. IfIsSuccessfulreturns true, the error is counted as a success. Otherwise the error is counted as a failure. IfIsSuccessfulis nil, defaultIsSuccessfulis used, which returns false for all non-nil errors.
The struct Counts holds the numbers of requests and their successes/failures:
type Counts struct {
Requests uint32
TotalSuccesses uint32
TotalFailures uint32
ConsecutiveSuccesses uint32
ConsecutiveFailures uint32
}
CircuitBreaker clears the internal Counts either
on the change of the state or at the closed-state intervals.
Counts ignores the results of the requests sent before clearing.
CircuitBreaker can wrap any function to send a request:
func (cb *CircuitBreaker) Execute(req func() (interface{}, error)) (interface{}, error)
The method Execute runs the given request if CircuitBreaker accepts it.
Execute returns an error instantly if CircuitBreaker rejects the request.
Otherwise, Execute returns the result of the request.
If a panic occurs in the request, CircuitBreaker handles it as an error
and causes the same panic again.
Example
var cb *breaker.CircuitBreaker
func Get(url string) ([]byte, error) {
body, err := cb.Execute(func() (interface{}, error) {
resp, err := http.Get(url)
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
return body, nil
})
if err != nil {
return nil, err
}
return body.([]byte), nil
}
See example for details.
License
The MIT License (MIT)
See LICENSE for details.