CHORUS/docs/development/sec-slurp-ucxl-beacon-pin-steward.md

SEC-SLURP UCXL Beacon & Pin Steward Design Notes

Purpose

• Establish the authoritative UCXL context beacon that bridges SLURP persistence with WHOOSH/role-aware agents.
• Define the Pin Steward responsibilities so DHT replication, healing, and telemetry satisfy SEC-SLURP 1.1a acceptance criteria.
• Provide an incremental execution plan aligned with the Persistence Wiring Report and DHT Resilience Supplement.

UCXL Beacon Data Model

• manifest_id (string): deterministic hash of project:task:address:version.
• ucxl_address (ucxl.Address): canonical address that produced the manifest.
• context_version (int): monotonic version from SLURP temporal graph.
• source_hash (string): content hash emitted by persistContext (LevelDB) for change detection.
• generated_by (string): CHORUS agent id / role bundle that wrote the context.
• generated_at (time.Time): timestamp from SLURP persistence event.
• replica_targets ([]string): desired replica node ids (Pin Steward enforces replication_factor).
• replica_state ([]ReplicaInfo): health snapshot (node_id, provider_id, status, last_checked, latency_ms).
• encryption (EncryptionMetadata):
  • dek_fingerprint (string)
  • kek_policy (string): BACKBEAT rotation policy identifier.
  • rotation_due (time.Time)
• compliance_tags ([]string): SHHH/WHOOSH governance hooks (e.g. sec-high, audit-required).
• beacon_metrics (BeaconMetrics): summarized counters for cache hits, DHT retrieves, validation errors.

Storage Strategy

• Primary persistence in LevelDB (pkg/slurp/slurp.go) using key prefix beacon::<manifest_id>.
• Secondary replication to DHT under dht://beacon/<manifest_id> enabling WHOOSH agents to read via Pin Steward API.
• Optional export to UCXL Decision Record envelope for historical traceability.

Beacon APIs

Endpoint	Purpose	Notes
Beacon.Upsert(manifest)	Persist/update manifest	Called by SLURP after persistContext success.
Beacon.Get(ucxlAddress)	Resolve latest manifest	Used by WHOOSH/agents to locate canonical context.
Beacon.List(filter)	Query manifests by tags/roles/time	Backs dashboards and Pin Steward audits.
Beacon.StreamChanges(since)	Provide change feed for Pin Steward anti-entropy jobs	Implements backpressure and bookmark tokens.

All APIs return envelope with UCXL citation + checksum to make SLURP⇄WHOOSH handoff auditable.

Pin Steward Responsibilities

1. Replication Planning
   • Read manifests via Beacon.StreamChanges.
   • Evaluate current replica_state vs. replication_factor from configuration.
   • Produce queue of DHT store/refresh tasks (storeAsync, storeSync, storeQuorum).
2. Healing & Anti-Entropy
   • Schedule heal_under_replicated jobs every anti_entropy_interval.
   • Re-announce providers on Pulse/Reverb when TTL < threshold.
   • Record outcomes back into manifest (replica_state).
3. Envelope Encryption Enforcement
   • Request KEK material from KACHING/SHHH as described in SEC-SLURP 1.1a.
   • Ensure DEK fingerprints match encryption metadata; trigger rotation if stale.
4. Telemetry Export
   • Emit Prometheus counters: pin_steward_replica_heal_total, pin_steward_replica_unhealthy, pin_steward_encryption_rotations_total.
   • Surface aggregated health to WHOOSH dashboards for council visibility.

Interaction Flow

1. SLURP Persistence
   • UpsertContext → LevelDB write → manifests assembled (persistContext).
   • Beacon Upsert called with manifest + context hash.
2. Pin Steward Intake
   • StreamChanges yields manifest → steward verifies encryption metadata and schedules replication tasks.
3. DHT Coordination
   • ReplicationManager.EnsureReplication invoked with target factor.
   • defaultVectorClockManager (temporary) to be replaced with libp2p-aware implementation for provider TTL tracking.
4. WHOOSH Consumption
   • WHOOSH SLURP proxy fetches manifest via Beacon.Get, caches in WHOOSH DB, attaches to deliverable artifacts.
   • Council UI surfaces replication state + encryption posture for operator decisions.

Incremental Delivery Plan

1. Sprint A (Persistence parity)
   • Finalize LevelDB manifest schema + tests (extend slurp_persistence_test.go).
   • Implement Beacon interfaces within SLURP service (in-memory + LevelDB).
   • Add Prometheus metrics for persistence reads/misses.
2. Sprint B (Pin Steward MVP)
   • Build steward worker with configurable reconciliation loop.
   • Wire to existing DistributedStorage stubs (StoreAsync/Sync/Quorum).
   • Emit health logs; integrate with CLI diagnostics.
3. Sprint C (DHT Resilience)
   • Swap defaultVectorClockManager with libp2p implementation; add provider TTL probes.
   • Implement envelope encryption path leveraging KACHING/SHHH interfaces (replace stubs in pkg/crypto).
   • Add CI checks: replica factor assertions, provider refresh tests, beacon schema validation.
4. Sprint D (WHOOSH Integration)
   • Expose REST/gRPC endpoint for WHOOSH to query manifests.
   • Update WHOOSH SLURPArtifactManager to require beacon confirmation before submission.
   • Surface Pin Steward alerts in WHOOSH admin UI.

Open Questions

• Confirm whether Beacon manifests should include DER signatures or rely on UCXL envelope hash.
• Determine storage for historical manifests (append-only log vs. latest-only) to support temporal rewind.
• Align Pin Steward job scheduling with existing BACKBEAT cadence to avoid conflicting rotations.

Next Actions

• Prototype BeaconStore interface + LevelDB implementation in SLURP package.
• Document Pin Steward anti-entropy algorithm with pseudocode and integrate into SEC-SLURP test plan.
• Sync with WHOOSH team on manifest query contract (REST vs. gRPC; pagination semantics).