# Backbeat Protocol — Pulse/Reverb (v0.1)

> **Purpose:** Give CHORUS a shared, lightweight rhythm so multi‑agent, p2p work doesn’t deadlock or drift. Standardise expectations (plan/work/review), exit conditions, promises, and time‑bounded collaboration across CHORUS, HMMM, SLURP, SHHH, UCXL, WHOOSH, and COOEE.

---

## 1) Rationale

- **Problem:** In pub/sub meshes, agents can wait indefinitely for help/context; there’s no universal cadence for planning, execution, or re‑evaluation.
    
- **Principle:** Use **coarse, explicit tempo** (beats/bars) for policy alignment; not for hard real‑time sync. Must be **partition‑tolerant**, **observable**, and **cheap**.
    
- **Design:** Human‑readable **beats/bars/phrases** for policy, **Hybrid Logical Clocks (HLC)** for mergeable ordering.
    

---

## 2) Core Concepts

- **Tempo (BPM):** Beats per minute (e.g., 6–30 BPM). Cluster‑level default; task classes may suggest hints.
    
- **Beat:** Base epoch (e.g., 4 s @ 15 BPM).
    
- **Bar:** Group of beats (e.g., 8). **Downbeat** (beat 1) is a soft barrier (checkpoints, secret rotation).
    
- **Phrase:** A sequence of bars that maps to a work cycle: **plan → work → review**.
    
- **Score (per task):** Declarative allocation of beats across phases + wait budgets + retries.
    

---

## 3) Roles & Components

- **Pulse:** Cluster tempo broadcaster. Publishes `BeatFrame` each beat; single elected leader (Raft/etcd), followers can degrade to local.
    
- **Reverb:** Aggregator/rollup. Ingests `StatusClaim`s and emits per‑bar `BarReport`s, plus hints for adaptive tempo.
    
- **Agents (CHORUS workers, HMMM collaborators, SLURP, etc.):** Consume beats, enforce **Score**, publish `StatusClaim`s.
    
- **SHHH:** Rotates short‑lived secrets **on downbeats** (per‑bar keys).
    
- **COOEE/DHT:** Transport for topics `backbeat://cluster/{id}` and per‑project status lanes.
    

### Implementation Snapshot (2025-10)

- **Pulse service (`cmd/pulse`)** – Encapsulates Raft leader election (`internal/backbeat/leader.go`), Hybrid Logical Clock maintenance (`internal/backbeat/hlc.go`), degradation control (`internal/backbeat/degradation.go`), and beat publishing over NATS. It also exposes an admin HTTP surface and collects tempo/drift metrics via `internal/backbeat/metrics.go`.
- **Reverb service (`cmd/reverb`)** – Subscribes to pulse beats and agent status subjects, aggregates `StatusClaim`s into rolling windows, and emits `BarReport`s on downbeats. Readiness, health, and Prometheus endpoints report claim throughput, aggregation latency, and NATs failures.
- **Go SDK (`pkg/sdk`)** – Provides clients for beat callbacks, status emission, and health reporting with retry/circuit breaker hooks. CHORUS (`project-queues/active/CHORUS/internal/backbeat/integration.go`) and WHOOSH (`project-queues/active/WHOOSH/internal/backbeat/integration.go`) embed the SDK to align runtime operations with cluster tempo.
- **Inter-module telemetry** – CHORUS maps P2P lifecycle operations (elections, DHT bootstrap, council delivery) into BACKBEAT status claims, while WHOOSH emits search/composer activity. This keeps Reverb windows authoritative for council health and informs SLURP/BUBBLE provenance.
- **Observability bundle** – Monitoring assets (`monitoring/`, `prometheus.yml`) plus service metrics export drift, tempo adjustments, Raft state, and window KPIs, meeting BACKBEAT-PER-001/002/003 targets and enabling WHOOSH scaling gates to react to rhythm degradation.

---

## 4) Wire Model

### 4.1 BeatFrame (Pulse → all)

```json
{
  "cluster_id": "chorus-aus-01",
  "tempo_bpm": 15,
  "beat_ms": 4000,
  "bar_len_beats": 8,
  "bar": 1287,
  "beat": 3,
  "phase": "work",
  "hlc": "2025-09-03T02:12:27.183Z+1287:3+17",
  "policy_hash": "sha256:...",
  "deadline_at": "2025-09-03T02:12:31.183Z"
}
```

### 4.2 StatusClaim (agents → Reverb)

```json
{
  "agent_id": "chorus-192-168-1-27",
  "task_id": "ucxl://...",
  "bar": 1287,
  "beat": 3,
  "state": "planning|executing|waiting|review|done|failed",
  "wait_for": ["hmmm://thread/abc"],
  "beats_left": 2,
  "progress": 0.42,
  "notes": "awaiting summarised artifact from peer",
  "hlc": "..."
}
```

### 4.3 HelpPromise (HMMM → requester)

```json
{
  "thread_id": "hmmm://thread/abc",
  "promise_beats": 2,
  "confidence": 0.7,
  "fail_after_beats": 3,
  "on_fail": "fallback-plan-A"
}
```

### 4.4 BarReport (Reverb → observability)

- Per‑bar rollup: task counts by state, overruns, broken promises, queue depth, utilisation hints, suggested tempo/phase tweak.
    

---

## 5) Score Spec (YAML)

```yaml
score:
  tempo: 15              # bpm hint; cluster policy can override
  bar_len: 8             # beats per bar
  phases:
    plan: 2              # beats
    work: 4
    review: 2
  wait_budget:
    help: 2              # max beats to wait for HMMM replies across the phrase
    io: 1                # max beats to wait for I/O
  retry:
    max_phrases: 2
    backoff: geometric   # plan/work/review shrink each retry
  escalation:
    on_wait_exhausted: ["emit:needs-attention", "fallback:coarse-answer"]
    on_overrun: ["checkpoint", "defer:next-phrase"]
```

> **Rule:** Agents must not exceed phase beat allocations. If `help` budget is exhausted, **exit cleanly** with degraded but auditable output.

---

## 6) Agent Loop (sketch)

```text
on BeatFrame(bf):
  if new bar and beat==1: rotate_ephemeral_keys(); checkpoint();

  phase = score.phase_for(bf.beat)

  switch phase:
    PLAN:
      if not planned: do_planning_until(phase_end)
    WORK:
      if need_help and !help_promised: request_help_with_promise()
      if waiting_for_help:
        if wait_beats > score.wait_budget.help: exit_with_fallback()
        else continue_work_on_alternative_path()
      else do_work_step()
    REVIEW:
      run_tests_and_summarise(); publish StatusClaim(state=done|failed)

  enforce_cutoffs_at_phase_boundaries()
```

---

## 7) Adaptive Tempo Controller (ATC)

- **Inputs:** Queue depth per role, GPU/CPU util (WHOOSH), overrun frequency, broken promises.
    
- **Policy:** Adjust `tempo_bpm` and/or redistribute phase beats **between bars only** (PI‑style control, hysteresis ±10%).
    
- **Guardrails:** ≤1 beat change per minute; freeze during incidents.
    

---

## 8) Exit Conditions & Deadlock Prevention

- **Wait budgets** are hard ceilings. Missing `HelpPromise` by end‑of‑bar triggers `on_wait_exhausted`.
    
- **Locks & leases** expire at bar boundaries unless renewed with `beats_left`.
    
- **Promises** include `promise_beats` and `fail_after_beats` so callers can plan.
    
- **Idempotent checkpoints** at downbeats enable safe retries/resumptions.
    

---

## 9) Integration Points

- **CHORUS (workers):** Consume `BeatFrame`; enforce `Score`; publish `StatusClaim` each beat/change.
    
- **HMMM (collab):** Replies carry `HelpPromise`; threads auto‑close if `fail_after_beats` elapses.
    
- **SLURP (curation):** Batch ingest windows tied to review beats; produce bar‑stamped artefacts.
    
- **SHHH (secrets):** Rotate per bar; credentials scoped to `<cluster,bar>`.
    
- **UCXL:** Attach tempo metadata to deliverables: `{bar, beat, hlc}`; optional address suffix `;bar=1287#beat=8`.
    
- **WHOOSH:** Expose utilisation to ATC; enforce resource leases in beat units.
    
- **COOEE/DHT:** Topics: `backbeat://cluster/{id}`, `status://{project}`, `promise://hmmm`.
    

---

## 10) Failure Modes & Degraded Operation

- **No Pulse leader:** Agents derive a **median‑of‑pulses** from available Purses; if none, use local monotonic clock (jitter ok) and **freeze tempo changes**.
    
- **Partitions:** Keep counting beats locally (HLC ensures mergeable order). Reverb reconciles by HLC and bar on heal.
    
- **Drift:** Tempo changes only on downbeats; publish `policy_hash` so agents detect misconfig.
    

---

## 11) Config Examples

### 11.1 Cluster Tempo Policy

```yaml
cluster_id: chorus-aus-01
initial_bpm: 12
bar_len_beats: 8
phases: [plan, work, review]
limits:
  max_bpm: 24
  min_bpm: 6
adaptation:
  enable: true
  hysteresis_pct: 10
  change_per_minute: 1_beat
observability:
  emit_bar_reports: true
```

### 11.2 Task Score (attached to UCXL deliverable)

```yaml
ucxl: ucxl://proj:any/*/task/graph_ingest
score:
  tempo: 15
  bar_len: 8
  phases: {plan: 2, work: 4, review: 2}
  wait_budget: {help: 2, io: 1}
  retry: {max_phrases: 2, backoff: geometric}
  escalation:
    on_wait_exhausted: ["emit:needs-attention", "fallback:coarse-answer"]
```

---

## 12) Observability

- **Per‑bar dashboards:** state counts, overruns, broken promises, tempo changes, queue depth, utilisation.
    
- **Trace stamps:** Every artifact/event carries `{bar, beat, hlc}` for forensic replay.
    
- **Alarms:** `promise_miss_rate`, `overrun_rate`, `no_status_claims`.
    

---

## 13) Security

- Rotate **ephemeral keys on downbeats**; scope to project/role when possible.
    
- Bar‑stamped tokens reduce blast radius; revoke at bar+N.
    

---

## 14) Economics & Budgeting — Beats as Unit of Cost

### 14.1 Beat Unit (BU)

- **Definition:** 1 BU = one cluster beat interval (`beat_ms`). It’s the atomic scheduling & accounting quantum.
    

### 14.2 Resource Primitives (WHOOSH‑measured)

- `cpu_sec`, `gpu_sec[class]`, `accel_sec[class]`, `mem_gbs` (GB·s), `disk_io_mb`, `net_egress_mb`, `storage_gbh`.
    

### 14.3 Budget & Costing

```yaml
budget:
  max_bu:  N_total
  phase_caps: { plan: Np, work: Nw, review: Nr }
  wait_caps:  { help: Nh, io: Ni }
  hard_end: bar+K
  charge_to: ucxl://acct/...
```

Cost per phrase:

```
Total = Σ(beats_used * role_rate_bu)
      + Σ_class(gpu_sec[class] * rate_gpu_sec[class])
      + cpu_sec*rate_cpu_sec + mem_gbs*rate_mem_gbs
      + disk_io_mb*rate_io_mb + net_egress_mb*rate_egress_mb
      + storage_gbh*rate_storage_gbh
```

### 14.4 KPIs

- **TNT** (tempo‑normalised throughput), **BPD** (beats per deliverable), **WR** (wait ratio), **η** (efficiency), **PMR** (promise miss rate), **CPD** (cost per deliverable), **TTFU** (time to first useful).
    

---

## 15) Tokenless Accounting (Hybrid CPU/GPU, On‑prem + Cloud)

- **No tokens.** Price **beats + measured resources**; ignore model‑token counts.
    
- **Device classes:** price per GPU/accelerator class (A100, 4090, MI300X, TPU…).
    
- **Rates:** on‑prem from TCO / duty‑cycle seconds; cloud from per‑second list prices. Bind via config.
    
- **Beat‑scoped caps:** per‑BU ceilings on resource primitives to contain spend regardless of hardware skew.
    
- **Calibration (planning‑only):** per‑family normalisers if you want **Effective Compute Units** for planning; **billing remains raw seconds**.
    

---

## 16) MVP Bring‑up Plan

1. **Pulse**: static BPM, broadcast `BeatFrame` over COOEE.
    
2. **Agents**: publish `StatusClaim`; enforce `wait_budget` & `HelpPromise`.
    
3. **Reverb**: roll up to `BarReport`; surface early KPIs.
    
4. **SHHH**: rotate credentials on downbeats.
    
5. **ATC**: enable adaptation after telemetry.
    

---

## 17) Open Questions

- Per‑role tempi vs one cluster tempo?
    
- Fixed `bar_len` vs dynamic redistribution of phase beats?
    
- Score UI: YAML + visual “score sheet” editor?
    

---

### Naming (on brand)

- **Backbeat Protocol** — **Pulse** (broadcaster) + **Reverb** (rollup & reports). Musical, expressive; conveys ripples from each downbeat.


# Backbeat — Relative Beats Addendum (UCXL ^^/~~)

**Why this addendum?** We’re removing dependence on ever‑increasing `bar`/`beat` counters. All coordination is expressed **relative to NOW** in **beats**, aligned with UCXL temporal markers `^^` (future) and `~~` (past).

## A) Wire Model Adjustments

### BeatFrame (Pulse → all)

**Replace** prior fields `{bar, beat}` with:

```json
{
  "cluster_id": "...",
  "tempo_bpm": 15,
  "beat_ms": 4000,
  "bar_len_beats": 8,
  "beat_index": 3,                  // 1..bar_len_beats (cyclic within bar)
  "beat_epoch": "2025-09-03T02:12:27.000Z", // start time of this beat
  "downbeat": false,                // true when beat_index==1
  "phase": "work",
  "hlc": "2025-09-03T02:12:27.183Z+17",
  "policy_hash": "sha256:...",
  "deadline_at": "2025-09-03T02:12:31.183Z"
}
```

### StatusClaim (agents → Reverb)

**Replace** prior fields `{bar, beat}` with:

```json
{
  "agent_id": "...",
  "task_id": "...",
  "beat_index": 3,
  "state": "planning|executing|waiting|review|done|failed",
  "beats_left": 2,
  "progress": 0.42,
  "notes": "...",
  "hlc": "..."
}
```

### Bar/Window Aggregation

- Reverb aggregates per **window** bounded by `downbeat=true` frames.
    
- **No global bar counters** are transmitted. Observability UIs may keep a local `window_id` for navigation.
    

## B) UCXL Temporal Suffix - (Requires RFC-UCXL 1.1)

Attach **relative beat** navigation to any UCXL address:

- `;beats=^^N` → target **N beats in the future** from now
    
- `;beats=~~N` → target **N beats in the past** from now
    
- Optional: `;phase=plan|work|review`
    

**Example:**

```
ucxl://proj:any/*/task/ingest;beats=^^4;phase=work
```

## C) Policy & Promises

- All time budgets are **Δbeats**: `wait_budget.help`, `retry.max_phrases`, `promise_beats`, `fail_after_beats`.
    
- **Leases/locks** renew per beat and expire on phase change unless renewed.
    

## D) Derivations

- `beat_index = 1 + floor( (unix_ms / beat_ms) mod bar_len_beats )` (derived locally).
    
- `beat_epoch = floor_to_multiple(now, beat_ms)`.
    
- `Δbeats(target_time) = round( (target_time - now) / beat_ms )`.
    

## E) Compatibility Notes

- Old fields `{bar, beat}` are **deprecated**; if received, they can be ignored or mapped to local windows.
    
- HLC remains the canonical merge key for causality.
    

## F) Action Items

1. Update the **spec wire model** sections accordingly.
    
2. Regenerate the **Go prototype** using `BeatIndex/BeatEpoch/Downbeat` instead of `Bar/Beat` counters.
    
3. Add UCXL parsing for `;beats=^^/~~` in RUSTLE.
- [ ] TODO: RUSTLE update for BACKBEAT compatibility