CHORUS/docs/development/sec-slurp-ucxl-beacon-pin-steward.md

# SEC-SLURP UCXL Beacon & Pin Steward Design Notes

## Purpose
- Establish the authoritative UCXL context beacon that bridges SLURP persistence with WHOOSH/role-aware agents.
- Define the Pin Steward responsibilities so DHT replication, healing, and telemetry satisfy SEC-SLURP 1.1a acceptance criteria.
- Provide an incremental execution plan aligned with the Persistence Wiring Report and DHT Resilience Supplement.

## UCXL Beacon Data Model
- **manifest_id** (`string`): deterministic hash of `project:task:address:version`.
- **ucxl_address** (`ucxl.Address`): canonical address that produced the manifest.
- **context_version** (`int`): monotonic version from SLURP temporal graph.
- **source_hash** (`string`): content hash emitted by `persistContext` (LevelDB) for change detection.
- **generated_by** (`string`): CHORUS agent id / role bundle that wrote the context.
- **generated_at** (`time.Time`): timestamp from SLURP persistence event.
- **replica_targets** (`[]string`): desired replica node ids (Pin Steward enforces `replication_factor`).
- **replica_state** (`[]ReplicaInfo`): health snapshot (`node_id`, `provider_id`, `status`, `last_checked`, `latency_ms`).
- **encryption** (`EncryptionMetadata`):
  - `dek_fingerprint` (`string`)
  - `kek_policy` (`string`): BACKBEAT rotation policy identifier.
  - `rotation_due` (`time.Time`)
- **compliance_tags** (`[]string`): SHHH/WHOOSH governance hooks (e.g. `sec-high`, `audit-required`).
- **beacon_metrics** (`BeaconMetrics`): summarized counters for cache hits, DHT retrieves, validation errors.

### Storage Strategy
- Primary persistence in LevelDB (`pkg/slurp/slurp.go`) using key prefix `beacon::<manifest_id>`.
- Secondary replication to DHT under `dht://beacon/<manifest_id>` enabling WHOOSH agents to read via Pin Steward API.
- Optional export to UCXL Decision Record envelope for historical traceability.

## Beacon APIs
| Endpoint | Purpose | Notes |
|----------|---------|-------|
| `Beacon.Upsert(manifest)` | Persist/update manifest | Called by SLURP after `persistContext` success. |
| `Beacon.Get(ucxlAddress)` | Resolve latest manifest | Used by WHOOSH/agents to locate canonical context. |
| `Beacon.List(filter)` | Query manifests by tags/roles/time | Backs dashboards and Pin Steward audits. |
| `Beacon.StreamChanges(since)` | Provide change feed for Pin Steward anti-entropy jobs | Implements backpressure and bookmark tokens. |

All APIs return envelope with UCXL citation + checksum to make SLURP⇄WHOOSH handoff auditable.

## Pin Steward Responsibilities
1. **Replication Planning**
   - Read manifests via `Beacon.StreamChanges`.
   - Evaluate current replica_state vs. `replication_factor` from configuration.
   - Produce queue of DHT store/refresh tasks (`storeAsync`, `storeSync`, `storeQuorum`).
2. **Healing & Anti-Entropy**
   - Schedule `heal_under_replicated` jobs every `anti_entropy_interval`.
   - Re-announce providers on Pulse/Reverb when TTL < threshold.
   - Record outcomes back into manifest (`replica_state`).
3. **Envelope Encryption Enforcement**
   - Request KEK material from KACHING/SHHH as described in SEC-SLURP 1.1a.
   - Ensure DEK fingerprints match `encryption` metadata; trigger rotation if stale.
4. **Telemetry Export**
   - Emit Prometheus counters: `pin_steward_replica_heal_total`, `pin_steward_replica_unhealthy`, `pin_steward_encryption_rotations_total`.
   - Surface aggregated health to WHOOSH dashboards for council visibility.

## Interaction Flow
1. **SLURP Persistence**
   - `UpsertContext` → LevelDB write → manifests assembled (`persistContext`).
   - Beacon `Upsert` called with manifest + context hash.
2. **Pin Steward Intake**
   - `StreamChanges` yields manifest → steward verifies encryption metadata and schedules replication tasks.
3. **DHT Coordination**
   - `ReplicationManager.EnsureReplication` invoked with target factor.
   - `defaultVectorClockManager` (temporary) to be replaced with libp2p-aware implementation for provider TTL tracking.
4. **WHOOSH Consumption**
   - WHOOSH SLURP proxy fetches manifest via `Beacon.Get`, caches in WHOOSH DB, attaches to deliverable artifacts.
   - Council UI surfaces replication state + encryption posture for operator decisions.

## Incremental Delivery Plan
1. **Sprint A (Persistence parity)**
   - Finalize LevelDB manifest schema + tests (extend `slurp_persistence_test.go`).
   - Implement Beacon interfaces within SLURP service (in-memory + LevelDB).
   - Add Prometheus metrics for persistence reads/misses.
2. **Sprint B (Pin Steward MVP)**
   - Build steward worker with configurable reconciliation loop.
   - Wire to existing `DistributedStorage` stubs (`StoreAsync/Sync/Quorum`).
   - Emit health logs; integrate with CLI diagnostics.
3. **Sprint C (DHT Resilience)**
   - Swap `defaultVectorClockManager` with libp2p implementation; add provider TTL probes.
   - Implement envelope encryption path leveraging KACHING/SHHH interfaces (replace stubs in `pkg/crypto`).
   - Add CI checks: replica factor assertions, provider refresh tests, beacon schema validation.
4. **Sprint D (WHOOSH Integration)**
   - Expose REST/gRPC endpoint for WHOOSH to query manifests.
   - Update WHOOSH SLURPArtifactManager to require beacon confirmation before submission.
   - Surface Pin Steward alerts in WHOOSH admin UI.

## Open Questions
- Confirm whether Beacon manifests should include DER signatures or rely on UCXL envelope hash.
- Determine storage for historical manifests (append-only log vs. latest-only) to support temporal rewind.
- Align Pin Steward job scheduling with existing BACKBEAT cadence to avoid conflicting rotations.

## Next Actions
- Prototype `BeaconStore` interface + LevelDB implementation in SLURP package.
- Document Pin Steward anti-entropy algorithm with pseudocode and integrate into SEC-SLURP test plan.
- Sync with WHOOSH team on manifest query contract (REST vs. gRPC; pagination semantics).