feat: Implement complete CHORUS leader election system
Major milestone: CHORUS leader election is now fully functional! ## Key Features Implemented: ### 🗳️ Leader Election Core - Fixed root cause: nodes now trigger elections when no admin exists - Added randomized election delays to prevent simultaneous elections - Implemented concurrent election prevention (only one election at a time) - Added proper election state management and transitions ### 📡 Admin Discovery System - Enhanced discovery requests with "WHOAMI" debug messages - Fixed discovery responses to properly include current leader ID - Added comprehensive discovery request/response logging - Implemented admin confirmation from multiple sources ### 🔧 Configuration Improvements - Increased discovery timeout from 3s to 15s for better reliability - Added proper Docker Hub image deployment workflow - Updated build process to use correct chorus-agent binary (not deprecated chorus) - Added static compilation flags for Alpine Linux compatibility ### 🐛 Critical Fixes - Fixed build process confusion between chorus vs chorus-agent binaries - Added missing admin_election capability to enable leader elections - Corrected discovery logic to handle zero admin responses - Enhanced debugging with detailed state and timing information ## Current Operational Status: ✅ Admin Election: Working with proper consensus ✅ Heartbeat System: 15-second intervals from elected admin ✅ Discovery Protocol: Nodes can find and confirm current admin ✅ P2P Connectivity: 5+ connected peers with libp2p ✅ SLURP Functionality: Enabled on admin nodes ✅ BACKBEAT Integration: Tempo synchronization working ✅ Container Health: All health checks passing ## Technical Details: - Election uses weighted scoring based on uptime, capabilities, and resources - Randomized delays prevent election storms (30-45s wait periods) - Discovery responses include current leader ID for network-wide consensus - State management prevents multiple concurrent elections - Enhanced logging provides full visibility into election process 🎉 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
132
vendor/github.com/sony/gobreaker/README.md
generated
vendored
Normal file
132
vendor/github.com/sony/gobreaker/README.md
generated
vendored
Normal file
@@ -0,0 +1,132 @@
|
||||
gobreaker
|
||||
=========
|
||||
|
||||
[](http://godoc.org/github.com/sony/gobreaker)
|
||||
|
||||
[gobreaker][repo-url] implements the [Circuit Breaker pattern](https://msdn.microsoft.com/en-us/library/dn589784.aspx) in Go.
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
```
|
||||
go get github.com/sony/gobreaker
|
||||
```
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
The struct `CircuitBreaker` is a state machine to prevent sending requests that are likely to fail.
|
||||
The function `NewCircuitBreaker` creates a new `CircuitBreaker`.
|
||||
|
||||
```go
|
||||
func NewCircuitBreaker(st Settings) *CircuitBreaker
|
||||
```
|
||||
|
||||
You can configure `CircuitBreaker` by the struct `Settings`:
|
||||
|
||||
```go
|
||||
type Settings struct {
|
||||
Name string
|
||||
MaxRequests uint32
|
||||
Interval time.Duration
|
||||
Timeout time.Duration
|
||||
ReadyToTrip func(counts Counts) bool
|
||||
OnStateChange func(name string, from State, to State)
|
||||
IsSuccessful func(err error) bool
|
||||
}
|
||||
```
|
||||
|
||||
- `Name` is the name of the `CircuitBreaker`.
|
||||
|
||||
- `MaxRequests` is the maximum number of requests allowed to pass through
|
||||
when the `CircuitBreaker` is half-open.
|
||||
If `MaxRequests` is 0, `CircuitBreaker` allows only 1 request.
|
||||
|
||||
- `Interval` is the cyclic period of the closed state
|
||||
for `CircuitBreaker` to clear the internal `Counts`, described later in this section.
|
||||
If `Interval` is 0, `CircuitBreaker` doesn't clear the internal `Counts` during the closed state.
|
||||
|
||||
- `Timeout` is the period of the open state,
|
||||
after which the state of `CircuitBreaker` becomes half-open.
|
||||
If `Timeout` is 0, the timeout value of `CircuitBreaker` is set to 60 seconds.
|
||||
|
||||
- `ReadyToTrip` is called with a copy of `Counts` whenever a request fails in the closed state.
|
||||
If `ReadyToTrip` returns true, `CircuitBreaker` will be placed into the open state.
|
||||
If `ReadyToTrip` is `nil`, default `ReadyToTrip` is used.
|
||||
Default `ReadyToTrip` returns true when the number of consecutive failures is more than 5.
|
||||
|
||||
- `OnStateChange` is called whenever the state of `CircuitBreaker` changes.
|
||||
|
||||
- `IsSuccessful` is called with the error returned from a request.
|
||||
If `IsSuccessful` returns true, the error is counted as a success.
|
||||
Otherwise the error is counted as a failure.
|
||||
If `IsSuccessful` is nil, default `IsSuccessful` is used, which returns false for all non-nil errors.
|
||||
|
||||
The struct `Counts` holds the numbers of requests and their successes/failures:
|
||||
|
||||
```go
|
||||
type Counts struct {
|
||||
Requests uint32
|
||||
TotalSuccesses uint32
|
||||
TotalFailures uint32
|
||||
ConsecutiveSuccesses uint32
|
||||
ConsecutiveFailures uint32
|
||||
}
|
||||
```
|
||||
|
||||
`CircuitBreaker` clears the internal `Counts` either
|
||||
on the change of the state or at the closed-state intervals.
|
||||
`Counts` ignores the results of the requests sent before clearing.
|
||||
|
||||
`CircuitBreaker` can wrap any function to send a request:
|
||||
|
||||
```go
|
||||
func (cb *CircuitBreaker) Execute(req func() (interface{}, error)) (interface{}, error)
|
||||
```
|
||||
|
||||
The method `Execute` runs the given request if `CircuitBreaker` accepts it.
|
||||
`Execute` returns an error instantly if `CircuitBreaker` rejects the request.
|
||||
Otherwise, `Execute` returns the result of the request.
|
||||
If a panic occurs in the request, `CircuitBreaker` handles it as an error
|
||||
and causes the same panic again.
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
```go
|
||||
var cb *breaker.CircuitBreaker
|
||||
|
||||
func Get(url string) ([]byte, error) {
|
||||
body, err := cb.Execute(func() (interface{}, error) {
|
||||
resp, err := http.Get(url)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
defer resp.Body.Close()
|
||||
body, err := ioutil.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return body, nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return body.([]byte), nil
|
||||
}
|
||||
```
|
||||
|
||||
See [example](https://github.com/sony/gobreaker/blob/master/example) for details.
|
||||
|
||||
License
|
||||
-------
|
||||
|
||||
The MIT License (MIT)
|
||||
|
||||
See [LICENSE](https://github.com/sony/gobreaker/blob/master/LICENSE) for details.
|
||||
|
||||
|
||||
[repo-url]: https://github.com/sony/gobreaker
|
||||
Reference in New Issue
Block a user