tony/CHORUS
1
0
Fork 0
Code Issues 10 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f7130b327c348c47456a90dc85b4aac3e8388c5e
CHORUS/docs/development/sec-slurp-ucxl-beacon-pin-steward.md
anthonyrawlins 4a77862289 chore: align slurp config and scaffolding
2025-09-27 21:03:12 +10:00

5.8 KiB
Raw Blame History

SEC-SLURP UCXL Beacon & Pin Steward Design Notes

Purpose

  • Establish the authoritative UCXL context beacon that bridges SLURP persistence with WHOOSH/role-aware agents.
  • Define the Pin Steward responsibilities so DHT replication, healing, and telemetry satisfy SEC-SLURP 1.1a acceptance criteria.
  • Provide an incremental execution plan aligned with the Persistence Wiring Report and DHT Resilience Supplement.

UCXL Beacon Data Model

  • manifest_id (string): deterministic hash of project:task:address:version.
  • ucxl_address (ucxl.Address): canonical address that produced the manifest.
  • context_version (int): monotonic version from SLURP temporal graph.
  • source_hash (string): content hash emitted by persistContext (LevelDB) for change detection.
  • generated_by (string): CHORUS agent id / role bundle that wrote the context.
  • generated_at (time.Time): timestamp from SLURP persistence event.
  • replica_targets ([]string): desired replica node ids (Pin Steward enforces replication_factor).
  • replica_state ([]ReplicaInfo): health snapshot (node_id, provider_id, status, last_checked, latency_ms).
  • encryption (EncryptionMetadata):
    • dek_fingerprint (string)
    • kek_policy (string): BACKBEAT rotation policy identifier.
    • rotation_due (time.Time)
  • compliance_tags ([]string): SHHH/WHOOSH governance hooks (e.g. sec-high, audit-required).
  • beacon_metrics (BeaconMetrics): summarized counters for cache hits, DHT retrieves, validation errors.

Storage Strategy

  • Primary persistence in LevelDB (pkg/slurp/slurp.go) using key prefix beacon::<manifest_id>.
  • Secondary replication to DHT under dht://beacon/<manifest_id> enabling WHOOSH agents to read via Pin Steward API.
  • Optional export to UCXL Decision Record envelope for historical traceability.

Beacon APIs

Endpoint Purpose Notes
Beacon.Upsert(manifest) Persist/update manifest Called by SLURP after persistContext success.
Beacon.Get(ucxlAddress) Resolve latest manifest Used by WHOOSH/agents to locate canonical context.
Beacon.List(filter) Query manifests by tags/roles/time Backs dashboards and Pin Steward audits.
Beacon.StreamChanges(since) Provide change feed for Pin Steward anti-entropy jobs Implements backpressure and bookmark tokens.

All APIs return envelope with UCXL citation + checksum to make SLURP⇄WHOOSH handoff auditable.

Pin Steward Responsibilities

  1. Replication Planning
    • Read manifests via Beacon.StreamChanges.
    • Evaluate current replica_state vs. replication_factor from configuration.
    • Produce queue of DHT store/refresh tasks (storeAsync, storeSync, storeQuorum).
  2. Healing & Anti-Entropy
    • Schedule heal_under_replicated jobs every anti_entropy_interval.
    • Re-announce providers on Pulse/Reverb when TTL < threshold.
    • Record outcomes back into manifest (replica_state).
  3. Envelope Encryption Enforcement
    • Request KEK material from KACHING/SHHH as described in SEC-SLURP 1.1a.
    • Ensure DEK fingerprints match encryption metadata; trigger rotation if stale.
  4. Telemetry Export
    • Emit Prometheus counters: pin_steward_replica_heal_total, pin_steward_replica_unhealthy, pin_steward_encryption_rotations_total.
    • Surface aggregated health to WHOOSH dashboards for council visibility.

Interaction Flow

  1. SLURP Persistence
    • UpsertContext → LevelDB write → manifests assembled (persistContext).
    • Beacon Upsert called with manifest + context hash.
  2. Pin Steward Intake
    • StreamChanges yields manifest → steward verifies encryption metadata and schedules replication tasks.
  3. DHT Coordination
    • ReplicationManager.EnsureReplication invoked with target factor.
    • defaultVectorClockManager (temporary) to be replaced with libp2p-aware implementation for provider TTL tracking.
  4. WHOOSH Consumption
    • WHOOSH SLURP proxy fetches manifest via Beacon.Get, caches in WHOOSH DB, attaches to deliverable artifacts.
    • Council UI surfaces replication state + encryption posture for operator decisions.

Incremental Delivery Plan

  1. Sprint A (Persistence parity)
    • Finalize LevelDB manifest schema + tests (extend slurp_persistence_test.go).
    • Implement Beacon interfaces within SLURP service (in-memory + LevelDB).
    • Add Prometheus metrics for persistence reads/misses.
  2. Sprint B (Pin Steward MVP)
    • Build steward worker with configurable reconciliation loop.
    • Wire to existing DistributedStorage stubs (StoreAsync/Sync/Quorum).
    • Emit health logs; integrate with CLI diagnostics.
  3. Sprint C (DHT Resilience)
    • Swap defaultVectorClockManager with libp2p implementation; add provider TTL probes.
    • Implement envelope encryption path leveraging KACHING/SHHH interfaces (replace stubs in pkg/crypto).
    • Add CI checks: replica factor assertions, provider refresh tests, beacon schema validation.
  4. Sprint D (WHOOSH Integration)
    • Expose REST/gRPC endpoint for WHOOSH to query manifests.
    • Update WHOOSH SLURPArtifactManager to require beacon confirmation before submission.
    • Surface Pin Steward alerts in WHOOSH admin UI.

Open Questions

  • Confirm whether Beacon manifests should include DER signatures or rely on UCXL envelope hash.
  • Determine storage for historical manifests (append-only log vs. latest-only) to support temporal rewind.
  • Align Pin Steward job scheduling with existing BACKBEAT cadence to avoid conflicting rotations.

Next Actions

  • Prototype BeaconStore interface + LevelDB implementation in SLURP package.
  • Document Pin Steward anti-entropy algorithm with pseudocode and integrate into SEC-SLURP test plan.
  • Sync with WHOOSH team on manifest query contract (REST vs. gRPC; pagination semantics).