Fix temporal persistence wiring and restore slurp_full suite

This commit is contained in:
anthonyrawlins
2025-09-28 11:39:03 +10:00
parent 9c32755632
commit 2ff408729c
16 changed files with 1195 additions and 802 deletions

View File

@@ -0,0 +1,20 @@
# Decision Record: Temporal Graph Persistence Integration
## Problem
Temporal graph nodes were only held in memory; the stub `persistTemporalNode` never touched the SEC-SLURP 1.1 persistence wiring or the context store. As a result, leader-elected agents could not rely on durable decision history and the write-buffer/replication mechanisms remained idle.
## Options Considered
1. **Leave persistence detached until the full storage stack ships.** Minimal work now, but temporal history would disappear on restart and the backlog of pending changes would grow untested.
2. **Wire the graph directly to the persistence manager and context store with sensible defaults.** Enables durability immediately, exercises the batch/flush pipeline, but requires choosing fallback role metadata for contexts that do not specify encryption targets.
## Decision
Adopt option 2. The temporal graph now forwards every node through the persistence manager (respecting the configured batch/flush behaviour) and synchronises the associated context via the `ContextStore` when role metadata is supplied. Default persistence settings guard against nil configuration, and the local storage layer now emits the shared `storage.ErrNotFound` sentinel for consistent error handling.
## Impact
- SEC-SLURP 1.1 write buffers and synchronization hooks are active, so leader nodes maintain durable temporal history.
- Context updates opportunistically reach the storage layer without blocking when role metadata is absent.
- Local storage consumers can reliably detect "not found" conditions via the new sentinel, simplifying mock alignment and future retries.
## Evidence
- Implemented in `pkg/slurp/temporal/graph_impl.go`, `pkg/slurp/temporal/persistence.go`, and `pkg/slurp/storage/local_storage.go`.
- Progress log: `docs/progress/report-SEC-SLURP-1.1.md`.

View File

@@ -0,0 +1,20 @@
# Decision Record: Temporal Package Stub Test Harness
## Problem
`GOWORK=off go test ./pkg/slurp/temporal` failed in the default build because the temporal tests exercised DHT/libp2p-dependent flows (graph compaction, influence analytics, navigator timelines). Without those providers, the suite crashed or asserted behaviour that the SEC-SLURP 1.1 stubs intentionally skip, blocking roadmap validation.
## Options Considered
1. **Re-implement the full temporal feature set against the new storage stubs now.** Pros: keeps existing high-value tests running. Cons: large scope, would delay the roadmap while the storage/index backlog is still unresolved.
2. **Disable or gate the expensive temporal suites and add a minimal stub-focused harness.** Pros: restores green builds quickly, isolates `slurp_full` coverage for when the heavy providers return, keeps feedback loop alive. Cons: reduces regression coverage in the default build until the full stack is back.
## Decision
Pursue option 2. Gate the original temporal integration/analytics tests behind the `slurp_full` build tag, introduce `pkg/slurp/temporal/temporal_stub_test.go` to exercise the stubbed lifecycle, and share helper scaffolding so both modes stay consistent. Align persistence helpers (`ContextStoreItem`, conflict resolution fields) and storage error contracts (`storage.ErrNotFound`) to keep the temporal package compiling in the stub build.
## Impact
- `GOWORK=off go test ./pkg/slurp/temporal` now passes in the default build, keeping SEC-SLURP 1.1 progress unblocked.
- The full temporal regression suite still runs when `-tags slurp_full` is supplied, preserving coverage for the production stack.
- Storage/persistence code now shares a sentinel error, reducing divergence between test doubles and future implementations.
## Evidence
- Code updates under `pkg/slurp/temporal/` and `pkg/slurp/storage/errors.go`.
- Progress log: `docs/progress/report-SEC-SLURP-1.1.md`.

View File

@@ -1,6 +1,10 @@
# SEC-SLURP 1.1 Persistence Wiring Report # SEC-SLURP 1.1 Persistence Wiring Report
## Summary of Changes ## Summary of Changes
- Restored the `slurp_full` temporal test suite by migrating influence adjacency across versions and cleaning compaction pruning to respect historical nodes.
- Connected the temporal graph to the persistence manager so new versions flush through the configured storage layers and update the context store when role metadata is available.
- Hardened the temporal package for the default build by aligning persistence helpers with the storage API (batch items now feed context payloads, conflict resolution fields match `types.go`), and by introducing a shared `storage.ErrNotFound` sentinel for mock stores and stub implementations.
- Gated the temporal integration/analysis suites behind the `slurp_full` build tag and added a lightweight stub test harness so `GOWORK=off go test ./pkg/slurp/temporal` runs cleanly without libp2p/DHT dependencies.
- Added LevelDB-backed persistence scaffolding in `pkg/slurp/slurp.go`, capturing the storage path, local storage handle, and the roadmap-tagged metrics helpers required for SEC-SLURP1.1. - Added LevelDB-backed persistence scaffolding in `pkg/slurp/slurp.go`, capturing the storage path, local storage handle, and the roadmap-tagged metrics helpers required for SEC-SLURP1.1.
- Upgraded SLURPs lifecycle so initialization bootstraps cached context data from disk, cache misses hydrate from persistence, successful `UpsertContext` calls write back to LevelDB, and shutdown closes the store with error telemetry. - Upgraded SLURPs lifecycle so initialization bootstraps cached context data from disk, cache misses hydrate from persistence, successful `UpsertContext` calls write back to LevelDB, and shutdown closes the store with error telemetry.
- Introduced `pkg/slurp/slurp_persistence_test.go` to confirm contexts survive process restarts and can be resolved after clearing in-memory caches. - Introduced `pkg/slurp/slurp_persistence_test.go` to confirm contexts survive process restarts and can be resolved after clearing in-memory caches.
@@ -12,6 +16,7 @@
- Attempted `GOWORK=off go test ./pkg/slurp`; the original authority-level blocker is resolved, but builds still fail in storage/index code due to remaining stub work (e.g., Bleve queries, DHT helpers). - Attempted `GOWORK=off go test ./pkg/slurp`; the original authority-level blocker is resolved, but builds still fail in storage/index code due to remaining stub work (e.g., Bleve queries, DHT helpers).
## Recommended Next Steps ## Recommended Next Steps
- Connect temporal persistence with the real distributed/DHT layers once available so sync/backup workers run against live replication targets.
- Stub the remaining storage/index dependencies (Bleve query scaffolding, UCXL helpers, `errorCh` queues, cache regex usage) or neutralize the heavy modules so that `GOWORK=off go test ./pkg/slurp` compiles and runs. - Stub the remaining storage/index dependencies (Bleve query scaffolding, UCXL helpers, `errorCh` queues, cache regex usage) or neutralize the heavy modules so that `GOWORK=off go test ./pkg/slurp` compiles and runs.
- Feed the durable store into the resolver and temporal graph implementations to finish the SEC-SLURP1.1 milestone once the package builds cleanly. - Feed the durable store into the resolver and temporal graph implementations to finish the SEC-SLURP1.1 milestone once the package builds cleanly.
- Extend Prometheus metrics/logging to track cache hit/miss ratios plus persistence errors for observability alignment. - Extend Prometheus metrics/logging to track cache hit/miss ratios plus persistence errors for observability alignment.

View File

@@ -0,0 +1,8 @@
package storage
import "errors"
// ErrNotFound indicates that the requested context does not exist in storage.
// Tests and higher-level components rely on this sentinel for consistent handling
// across local, distributed, and encrypted backends.
var ErrNotFound = errors.New("storage: not found")

View File

@@ -201,7 +201,7 @@ func (ls *LocalStorageImpl) Retrieve(ctx context.Context, key string) (interface
entryBytes, err := ls.db.Get([]byte(key), nil) entryBytes, err := ls.db.Get([]byte(key), nil)
if err != nil { if err != nil {
if err == leveldb.ErrNotFound { if err == leveldb.ErrNotFound {
return nil, fmt.Errorf("key not found: %s", key) return nil, fmt.Errorf("%w: %s", ErrNotFound, key)
} }
return nil, fmt.Errorf("failed to retrieve data: %w", err) return nil, fmt.Errorf("failed to retrieve data: %w", err)
} }
@@ -328,7 +328,7 @@ func (ls *LocalStorageImpl) Size(ctx context.Context, key string) (int64, error)
entryBytes, err := ls.db.Get([]byte(key), nil) entryBytes, err := ls.db.Get([]byte(key), nil)
if err != nil { if err != nil {
if err == leveldb.ErrNotFound { if err == leveldb.ErrNotFound {
return 0, fmt.Errorf("key not found: %s", key) return 0, fmt.Errorf("%w: %s", ErrNotFound, key)
} }
return 0, fmt.Errorf("failed to get data size: %w", err) return 0, fmt.Errorf("failed to get data size: %w", err)
} }

View File

@@ -20,6 +20,7 @@ type temporalGraphImpl struct {
// Core storage // Core storage
storage storage.ContextStore storage storage.ContextStore
persistence nodePersister
// In-memory graph structures for fast access // In-memory graph structures for fast access
nodes map[string]*TemporalNode // nodeID -> TemporalNode nodes map[string]*TemporalNode // nodeID -> TemporalNode
@@ -42,6 +43,10 @@ type temporalGraphImpl struct {
stalenessWeight *StalenessWeights stalenessWeight *StalenessWeights
} }
type nodePersister interface {
PersistTemporalNode(ctx context.Context, node *TemporalNode) error
}
// NewTemporalGraph creates a new temporal graph implementation // NewTemporalGraph creates a new temporal graph implementation
func NewTemporalGraph(storage storage.ContextStore) TemporalGraph { func NewTemporalGraph(storage storage.ContextStore) TemporalGraph {
return &temporalGraphImpl{ return &temporalGraphImpl{
@@ -177,16 +182,40 @@ func (tg *temporalGraphImpl) EvolveContext(ctx context.Context, address ucxl.Add
} }
// Copy influence relationships from parent // Copy influence relationships from parent
if len(latestNode.Influences) > 0 {
temporalNode.Influences = append([]ucxl.Address(nil), latestNode.Influences...)
} else {
temporalNode.Influences = make([]ucxl.Address, 0)
}
if len(latestNode.InfluencedBy) > 0 {
temporalNode.InfluencedBy = append([]ucxl.Address(nil), latestNode.InfluencedBy...)
} else {
temporalNode.InfluencedBy = make([]ucxl.Address, 0)
}
if latestNodeInfluences, exists := tg.influences[latestNode.ID]; exists { if latestNodeInfluences, exists := tg.influences[latestNode.ID]; exists {
tg.influences[nodeID] = make([]string, len(latestNodeInfluences)) cloned := append([]string(nil), latestNodeInfluences...)
copy(tg.influences[nodeID], latestNodeInfluences) tg.influences[nodeID] = cloned
for _, targetID := range cloned {
tg.influencedBy[targetID] = ensureString(tg.influencedBy[targetID], nodeID)
if targetNode, ok := tg.nodes[targetID]; ok {
targetNode.InfluencedBy = ensureAddress(targetNode.InfluencedBy, address)
}
}
} else { } else {
tg.influences[nodeID] = make([]string, 0) tg.influences[nodeID] = make([]string, 0)
} }
if latestNodeInfluencedBy, exists := tg.influencedBy[latestNode.ID]; exists { if latestNodeInfluencedBy, exists := tg.influencedBy[latestNode.ID]; exists {
tg.influencedBy[nodeID] = make([]string, len(latestNodeInfluencedBy)) cloned := append([]string(nil), latestNodeInfluencedBy...)
copy(tg.influencedBy[nodeID], latestNodeInfluencedBy) tg.influencedBy[nodeID] = cloned
for _, sourceID := range cloned {
tg.influences[sourceID] = ensureString(tg.influences[sourceID], nodeID)
if sourceNode, ok := tg.nodes[sourceID]; ok {
sourceNode.Influences = ensureAddress(sourceNode.Influences, address)
}
}
} else { } else {
tg.influencedBy[nodeID] = make([]string, 0) tg.influencedBy[nodeID] = make([]string, 0)
} }
@@ -534,8 +563,7 @@ func (tg *temporalGraphImpl) FindDecisionPath(ctx context.Context, from, to ucxl
return nil, fmt.Errorf("from node not found: %w", err) return nil, fmt.Errorf("from node not found: %w", err)
} }
_, err := tg.getLatestNodeUnsafe(to) if _, err := tg.getLatestNodeUnsafe(to); err != nil {
if err != nil {
return nil, fmt.Errorf("to node not found: %w", err) return nil, fmt.Errorf("to node not found: %w", err)
} }
@@ -750,31 +778,73 @@ func (tg *temporalGraphImpl) CompactHistory(ctx context.Context, beforeTime time
compacted := 0 compacted := 0
// For each address, keep only the latest version and major milestones before the cutoff
for address, nodes := range tg.addressToNodes { for address, nodes := range tg.addressToNodes {
toKeep := make([]*TemporalNode, 0) if len(nodes) == 0 {
continue
}
latestNode := nodes[len(nodes)-1]
toKeep := make([]*TemporalNode, 0, len(nodes))
toRemove := make([]*TemporalNode, 0) toRemove := make([]*TemporalNode, 0)
for _, node := range nodes { for _, node := range nodes {
// Always keep nodes after the cutoff time if node == latestNode {
if node.Timestamp.After(beforeTime) {
toKeep = append(toKeep, node) toKeep = append(toKeep, node)
continue continue
} }
// Keep major changes and influential decisions if node.Timestamp.After(beforeTime) || tg.isMajorChange(node) || tg.isInfluentialDecision(node) {
if tg.isMajorChange(node) || tg.isInfluentialDecision(node) {
toKeep = append(toKeep, node) toKeep = append(toKeep, node)
} else { continue
}
toRemove = append(toRemove, node) toRemove = append(toRemove, node)
} }
if len(toKeep) == 0 {
toKeep = append(toKeep, latestNode)
} }
// Update the address mapping sort.Slice(toKeep, func(i, j int) bool {
return toKeep[i].Version < toKeep[j].Version
})
tg.addressToNodes[address] = toKeep tg.addressToNodes[address] = toKeep
// Remove old nodes from main maps
for _, node := range toRemove { for _, node := range toRemove {
if outgoing, exists := tg.influences[node.ID]; exists {
for _, targetID := range outgoing {
tg.influencedBy[targetID] = tg.removeFromSlice(tg.influencedBy[targetID], node.ID)
if targetNode, ok := tg.nodes[targetID]; ok {
targetNode.InfluencedBy = tg.removeAddressFromSlice(targetNode.InfluencedBy, node.UCXLAddress)
}
}
}
if incoming, exists := tg.influencedBy[node.ID]; exists {
for _, sourceID := range incoming {
tg.influences[sourceID] = tg.removeFromSlice(tg.influences[sourceID], node.ID)
if sourceNode, ok := tg.nodes[sourceID]; ok {
sourceNode.Influences = tg.removeAddressFromSlice(sourceNode.Influences, node.UCXLAddress)
}
}
}
if decisionNodes, exists := tg.decisionToNodes[node.DecisionID]; exists {
filtered := make([]*TemporalNode, 0, len(decisionNodes))
for _, candidate := range decisionNodes {
if candidate.ID != node.ID {
filtered = append(filtered, candidate)
}
}
if len(filtered) == 0 {
delete(tg.decisionToNodes, node.DecisionID)
delete(tg.decisions, node.DecisionID)
} else {
tg.decisionToNodes[node.DecisionID] = filtered
}
}
delete(tg.nodes, node.ID) delete(tg.nodes, node.ID)
delete(tg.influences, node.ID) delete(tg.influences, node.ID)
delete(tg.influencedBy, node.ID) delete(tg.influencedBy, node.ID)
@@ -782,7 +852,6 @@ func (tg *temporalGraphImpl) CompactHistory(ctx context.Context, beforeTime time
} }
} }
// Clear caches after compaction
tg.pathCache = make(map[string][]*DecisionStep) tg.pathCache = make(map[string][]*DecisionStep)
tg.metricsCache = make(map[string]interface{}) tg.metricsCache = make(map[string]interface{})
@@ -901,10 +970,60 @@ func (tg *temporalGraphImpl) isInfluentialDecision(node *TemporalNode) bool {
} }
func (tg *temporalGraphImpl) persistTemporalNode(ctx context.Context, node *TemporalNode) error { func (tg *temporalGraphImpl) persistTemporalNode(ctx context.Context, node *TemporalNode) error {
// Convert to storage format and persist if node == nil {
// This would integrate with the storage system return fmt.Errorf("temporal node cannot be nil")
// For now, we'll assume persistence happens in memory }
if tg.persistence != nil {
if err := tg.persistence.PersistTemporalNode(ctx, node); err != nil {
return fmt.Errorf("failed to persist temporal node: %w", err)
}
}
if tg.storage == nil || node.Context == nil {
return nil return nil
}
roles := node.Context.EncryptedFor
if len(roles) == 0 {
roles = []string{"default"}
}
exists, err := tg.storage.ExistsContext(ctx, node.Context.UCXLAddress)
if err != nil {
return fmt.Errorf("failed to check context existence: %w", err)
}
if exists {
if err := tg.storage.UpdateContext(ctx, node.Context, roles); err != nil {
return fmt.Errorf("failed to update context for %s: %w", node.Context.UCXLAddress.String(), err)
}
return nil
}
if err := tg.storage.StoreContext(ctx, node.Context, roles); err != nil {
return fmt.Errorf("failed to store context for %s: %w", node.Context.UCXLAddress.String(), err)
}
return nil
}
func ensureString(list []string, value string) []string {
for _, existing := range list {
if existing == value {
return list
}
}
return append(list, value)
}
func ensureAddress(list []ucxl.Address, value ucxl.Address) []ucxl.Address {
for _, existing := range list {
if existing.String() == value.String() {
return list
}
}
return append(list, value)
} }
func contains(s, substr string) bool { func contains(s, substr string) bool {

View File

@@ -1,131 +1,23 @@
//go:build slurp_full
// +build slurp_full
package temporal package temporal
import ( import (
"context" "context"
"fmt"
"testing" "testing"
"time" "time"
"chorus/pkg/ucxl"
slurpContext "chorus/pkg/slurp/context" slurpContext "chorus/pkg/slurp/context"
"chorus/pkg/slurp/storage" "chorus/pkg/ucxl"
) )
// Mock storage for testing
type mockStorage struct {
data map[string]interface{}
}
func newMockStorage() *mockStorage {
return &mockStorage{
data: make(map[string]interface{}),
}
}
func (ms *mockStorage) StoreContext(ctx context.Context, node *slurpContext.ContextNode, roles []string) error {
ms.data[node.UCXLAddress.String()] = node
return nil
}
func (ms *mockStorage) RetrieveContext(ctx context.Context, address ucxl.Address, role string) (*slurpContext.ContextNode, error) {
if data, exists := ms.data[address.String()]; exists {
return data.(*slurpContext.ContextNode), nil
}
return nil, storage.ErrNotFound
}
func (ms *mockStorage) UpdateContext(ctx context.Context, node *slurpContext.ContextNode, roles []string) error {
ms.data[node.UCXLAddress.String()] = node
return nil
}
func (ms *mockStorage) DeleteContext(ctx context.Context, address ucxl.Address) error {
delete(ms.data, address.String())
return nil
}
func (ms *mockStorage) ExistsContext(ctx context.Context, address ucxl.Address) (bool, error) {
_, exists := ms.data[address.String()]
return exists, nil
}
func (ms *mockStorage) ListContexts(ctx context.Context, criteria *storage.ListCriteria) ([]*slurpContext.ContextNode, error) {
results := make([]*slurpContext.ContextNode, 0)
for _, data := range ms.data {
if node, ok := data.(*slurpContext.ContextNode); ok {
results = append(results, node)
}
}
return results, nil
}
func (ms *mockStorage) SearchContexts(ctx context.Context, query *storage.SearchQuery) (*storage.SearchResults, error) {
return &storage.SearchResults{}, nil
}
func (ms *mockStorage) BatchStore(ctx context.Context, batch *storage.BatchStoreRequest) (*storage.BatchStoreResult, error) {
return &storage.BatchStoreResult{}, nil
}
func (ms *mockStorage) BatchRetrieve(ctx context.Context, batch *storage.BatchRetrieveRequest) (*storage.BatchRetrieveResult, error) {
return &storage.BatchRetrieveResult{}, nil
}
func (ms *mockStorage) GetStorageStats(ctx context.Context) (*storage.StorageStatistics, error) {
return &storage.StorageStatistics{}, nil
}
func (ms *mockStorage) Sync(ctx context.Context) error {
return nil
}
func (ms *mockStorage) Backup(ctx context.Context, destination string) error {
return nil
}
func (ms *mockStorage) Restore(ctx context.Context, source string) error {
return nil
}
// Test helpers
func createTestAddress(path string) ucxl.Address {
addr, _ := ucxl.ParseAddress(fmt.Sprintf("ucxl://test/%s", path))
return *addr
}
func createTestContext(path string, technologies []string) *slurpContext.ContextNode {
return &slurpContext.ContextNode{
Path: path,
UCXLAddress: createTestAddress(path),
Summary: fmt.Sprintf("Test context for %s", path),
Purpose: fmt.Sprintf("Test purpose for %s", path),
Technologies: technologies,
Tags: []string{"test"},
Insights: []string{"test insight"},
GeneratedAt: time.Now(),
RAGConfidence: 0.8,
}
}
func createTestDecision(id, maker, rationale string, scope ImpactScope) *DecisionMetadata {
return &DecisionMetadata{
ID: id,
Maker: maker,
Rationale: rationale,
Scope: scope,
ConfidenceLevel: 0.8,
ExternalRefs: []string{},
CreatedAt: time.Now(),
ImplementationStatus: "complete",
Metadata: make(map[string]interface{}),
}
}
// Core temporal graph tests // Core temporal graph tests
func TestTemporalGraph_CreateInitialContext(t *testing.T) { func TestTemporalGraph_CreateInitialContext(t *testing.T) {
storage := newMockStorage() storage := newMockStorage()
graph := NewTemporalGraph(storage) graph := NewTemporalGraph(storage).(*temporalGraphImpl)
ctx := context.Background() ctx := context.Background()
address := createTestAddress("test/component") address := createTestAddress("test/component")
@@ -478,14 +370,14 @@ func TestTemporalGraph_ValidateIntegrity(t *testing.T) {
func TestTemporalGraph_CompactHistory(t *testing.T) { func TestTemporalGraph_CompactHistory(t *testing.T) {
storage := newMockStorage() storage := newMockStorage()
graph := NewTemporalGraph(storage) graphBase := NewTemporalGraph(storage)
graph := graphBase.(*temporalGraphImpl)
ctx := context.Background() ctx := context.Background()
address := createTestAddress("test/component") address := createTestAddress("test/component")
initialContext := createTestContext("test/component", []string{"go"}) initialContext := createTestContext("test/component", []string{"go"})
// Create initial version (old) // Create initial version (old)
oldTime := time.Now().Add(-60 * 24 * time.Hour) // 60 days ago
_, err := graph.CreateInitialContext(ctx, address, initialContext, "test_creator") _, err := graph.CreateInitialContext(ctx, address, initialContext, "test_creator")
if err != nil { if err != nil {
t.Fatalf("Failed to create initial context: %v", err) t.Fatalf("Failed to create initial context: %v", err)
@@ -510,6 +402,13 @@ func TestTemporalGraph_CompactHistory(t *testing.T) {
} }
} }
// Mark older versions beyond the retention window
for _, node := range graph.addressToNodes[address.String()] {
if node.Version <= 6 {
node.Timestamp = time.Now().Add(-60 * 24 * time.Hour)
}
}
// Get history before compaction // Get history before compaction
historyBefore, err := graph.GetEvolutionHistory(ctx, address) historyBefore, err := graph.GetEvolutionHistory(ctx, address)
if err != nil { if err != nil {

View File

@@ -899,14 +899,15 @@ func (ia *influenceAnalyzerImpl) findShortestPathLength(fromID, toID string) int
func (ia *influenceAnalyzerImpl) getNodeCentrality(nodeID string) float64 { func (ia *influenceAnalyzerImpl) getNodeCentrality(nodeID string) float64 {
// Simple centrality based on degree // Simple centrality based on degree
influencedBy := len(ia.graph.influencedBy[nodeID]) outgoing := len(ia.graph.influences[nodeID])
incoming := len(ia.graph.influencedBy[nodeID])
totalNodes := len(ia.graph.nodes) totalNodes := len(ia.graph.nodes)
if totalNodes <= 1 { if totalNodes <= 1 {
return 0 return 0
} }
return float64(influences+influencedBy) / float64(totalNodes-1) return float64(outgoing+incoming) / float64(totalNodes-1)
} }
func (ia *influenceAnalyzerImpl) calculateNodeDegreeCentrality(nodeID string) float64 { func (ia *influenceAnalyzerImpl) calculateNodeDegreeCentrality(nodeID string) float64 {
@@ -968,7 +969,6 @@ func (ia *influenceAnalyzerImpl) calculateNodeClosenessCentrality(nodeID string)
func (ia *influenceAnalyzerImpl) calculateNodePageRank(nodeID string) float64 { func (ia *influenceAnalyzerImpl) calculateNodePageRank(nodeID string) float64 {
// This is already calculated in calculatePageRank, so we'll use a simple approximation // This is already calculated in calculatePageRank, so we'll use a simple approximation
influences := len(ia.graph.influences[nodeID])
influencedBy := len(ia.graph.influencedBy[nodeID]) influencedBy := len(ia.graph.influencedBy[nodeID])
// Simple approximation based on in-degree with damping // Simple approximation based on in-degree with damping

View File

@@ -1,12 +1,16 @@
//go:build slurp_full
// +build slurp_full
package temporal package temporal
import ( import (
"context" "context"
"fmt"
"testing" "testing"
"time" "time"
"chorus/pkg/ucxl"
slurpContext "chorus/pkg/slurp/context" slurpContext "chorus/pkg/slurp/context"
"chorus/pkg/ucxl"
) )
func TestInfluenceAnalyzer_AnalyzeInfluenceNetwork(t *testing.T) { func TestInfluenceAnalyzer_AnalyzeInfluenceNetwork(t *testing.T) {
@@ -322,7 +326,6 @@ func TestInfluenceAnalyzer_PredictInfluence(t *testing.T) {
// Should predict influence to service2 (similar tech stack) // Should predict influence to service2 (similar tech stack)
foundService2 := false foundService2 := false
foundService3 := false
for _, prediction := range predictions { for _, prediction := range predictions {
if prediction.To.String() == addr2.String() { if prediction.To.String() == addr2.String() {
@@ -332,9 +335,6 @@ func TestInfluenceAnalyzer_PredictInfluence(t *testing.T) {
t.Errorf("Expected higher prediction probability for similar service, got %f", prediction.Probability) t.Errorf("Expected higher prediction probability for similar service, got %f", prediction.Probability)
} }
} }
if prediction.To.String() == addr3.String() {
foundService3 = true
}
} }
if !foundService2 && len(predictions) > 0 { if !foundService2 && len(predictions) > 0 {

View File

@@ -1,13 +1,17 @@
//go:build slurp_full
// +build slurp_full
package temporal package temporal
import ( import (
"context" "context"
"fmt"
"testing" "testing"
"time" "time"
"chorus/pkg/ucxl"
slurpContext "chorus/pkg/slurp/context" slurpContext "chorus/pkg/slurp/context"
"chorus/pkg/slurp/storage" "chorus/pkg/slurp/storage"
"chorus/pkg/ucxl"
) )
// Integration tests for the complete temporal graph system // Integration tests for the complete temporal graph system
@@ -723,7 +727,6 @@ func (m *mockBackupManager) CreateBackup(ctx context.Context, config *storage.Ba
ID: "test-backup-1", ID: "test-backup-1",
CreatedAt: time.Now(), CreatedAt: time.Now(),
Size: 1024, Size: 1024,
Description: "Test backup",
}, nil }, nil
} }

View File

@@ -62,8 +62,19 @@ func (dn *decisionNavigatorImpl) NavigateDecisionHops(ctx context.Context, addre
dn.mu.RLock() dn.mu.RLock()
defer dn.mu.RUnlock() defer dn.mu.RUnlock()
// Get starting node // Determine starting node based on navigation direction
startNode, err := dn.graph.getLatestNodeUnsafe(address) var (
startNode *TemporalNode
err error
)
switch direction {
case NavigationForward:
startNode, err = dn.graph.GetVersionAtDecision(ctx, address, 1)
default:
startNode, err = dn.graph.getLatestNodeUnsafe(address)
}
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to get starting node: %w", err) return nil, fmt.Errorf("failed to get starting node: %w", err)
} }
@@ -254,9 +265,7 @@ func (dn *decisionNavigatorImpl) ResetNavigation(ctx context.Context, address uc
// Clear any navigation sessions for this address // Clear any navigation sessions for this address
for _, session := range dn.navigationSessions { for _, session := range dn.navigationSessions {
if session.CurrentPosition.String() == address.String() { if session.CurrentPosition.String() == address.String() {
// Reset to latest version if _, err := dn.graph.getLatestNodeUnsafe(address); err != nil {
latestNode, err := dn.graph.getLatestNodeUnsafe(address)
if err != nil {
return fmt.Errorf("failed to get latest node: %w", err) return fmt.Errorf("failed to get latest node: %w", err)
} }

View File

@@ -1,12 +1,14 @@
//go:build slurp_full
// +build slurp_full
package temporal package temporal
import ( import (
"context" "context"
"fmt"
"testing" "testing"
"time"
"chorus/pkg/ucxl" "chorus/pkg/ucxl"
slurpContext "chorus/pkg/slurp/context"
) )
func TestDecisionNavigator_NavigateDecisionHops(t *testing.T) { func TestDecisionNavigator_NavigateDecisionHops(t *testing.T) {
@@ -36,7 +38,7 @@ func TestDecisionNavigator_NavigateDecisionHops(t *testing.T) {
} }
// Test forward navigation from version 1 // Test forward navigation from version 1
v1, err := graph.GetVersionAtDecision(ctx, address, 1) _, err = graph.GetVersionAtDecision(ctx, address, 1)
if err != nil { if err != nil {
t.Fatalf("Failed to get version 1: %v", err) t.Fatalf("Failed to get version 1: %v", err)
} }
@@ -371,7 +373,7 @@ func BenchmarkDecisionNavigator_FindStaleContexts(b *testing.B) {
graph.mu.Lock() graph.mu.Lock()
for _, nodes := range graph.addressToNodes { for _, nodes := range graph.addressToNodes {
for _, node := range nodes { for _, node := range nodes {
node.Staleness = 0.3 + (float64(node.Version)*0.1) // Varying staleness node.Staleness = 0.3 + (float64(node.Version) * 0.1) // Varying staleness
} }
} }
graph.mu.Unlock() graph.mu.Unlock()

View File

@@ -8,7 +8,6 @@ import (
"time" "time"
"chorus/pkg/slurp/storage" "chorus/pkg/slurp/storage"
"chorus/pkg/ucxl"
) )
// persistenceManagerImpl handles persistence and synchronization of temporal graph data // persistenceManagerImpl handles persistence and synchronization of temporal graph data
@@ -151,6 +150,8 @@ func NewPersistenceManager(
config *PersistenceConfig, config *PersistenceConfig,
) *persistenceManagerImpl { ) *persistenceManagerImpl {
cfg := normalizePersistenceConfig(config)
pm := &persistenceManagerImpl{ pm := &persistenceManagerImpl{
contextStore: contextStore, contextStore: contextStore,
localStorage: localStorage, localStorage: localStorage,
@@ -158,30 +159,96 @@ func NewPersistenceManager(
encryptedStore: encryptedStore, encryptedStore: encryptedStore,
backupManager: backupManager, backupManager: backupManager,
graph: graph, graph: graph,
config: config, config: cfg,
pendingChanges: make(map[string]*PendingChange), pendingChanges: make(map[string]*PendingChange),
conflictResolver: NewDefaultConflictResolver(), conflictResolver: NewDefaultConflictResolver(),
batchSize: config.BatchSize, batchSize: cfg.BatchSize,
writeBuffer: make([]*TemporalNode, 0, config.BatchSize), writeBuffer: make([]*TemporalNode, 0, cfg.BatchSize),
flushInterval: config.FlushInterval, flushInterval: cfg.FlushInterval,
}
if graph != nil {
graph.persistence = pm
} }
// Start background processes // Start background processes
if config.EnableAutoSync { if cfg.EnableAutoSync {
go pm.syncWorker() go pm.syncWorker()
} }
if config.EnableWriteBuffer { if cfg.EnableWriteBuffer {
go pm.flushWorker() go pm.flushWorker()
} }
if config.EnableAutoBackup { if cfg.EnableAutoBackup {
go pm.backupWorker() go pm.backupWorker()
} }
return pm return pm
} }
func normalizePersistenceConfig(config *PersistenceConfig) *PersistenceConfig {
if config == nil {
return defaultPersistenceConfig()
}
cloned := *config
if cloned.BatchSize <= 0 {
cloned.BatchSize = 1
}
if cloned.FlushInterval <= 0 {
cloned.FlushInterval = 30 * time.Second
}
if cloned.SyncInterval <= 0 {
cloned.SyncInterval = 15 * time.Minute
}
if cloned.MaxSyncRetries <= 0 {
cloned.MaxSyncRetries = 3
}
if len(cloned.EncryptionRoles) == 0 {
cloned.EncryptionRoles = []string{"default"}
} else {
cloned.EncryptionRoles = append([]string(nil), cloned.EncryptionRoles...)
}
if cloned.KeyPrefix == "" {
cloned.KeyPrefix = "temporal_graph"
}
if cloned.NodeKeyPattern == "" {
cloned.NodeKeyPattern = "temporal_graph/nodes/%s"
}
if cloned.GraphKeyPattern == "" {
cloned.GraphKeyPattern = "temporal_graph/graph/%s"
}
if cloned.MetadataKeyPattern == "" {
cloned.MetadataKeyPattern = "temporal_graph/metadata/%s"
}
return &cloned
}
func defaultPersistenceConfig() *PersistenceConfig {
return &PersistenceConfig{
EnableLocalStorage: true,
EnableDistributedStorage: false,
EnableEncryption: false,
EncryptionRoles: []string{"default"},
SyncInterval: 15 * time.Minute,
ConflictResolutionStrategy: "latest_wins",
EnableAutoSync: false,
MaxSyncRetries: 3,
BatchSize: 1,
FlushInterval: 30 * time.Second,
EnableWriteBuffer: false,
EnableAutoBackup: false,
BackupInterval: 24 * time.Hour,
RetainBackupCount: 3,
KeyPrefix: "temporal_graph",
NodeKeyPattern: "temporal_graph/nodes/%s",
GraphKeyPattern: "temporal_graph/graph/%s",
MetadataKeyPattern: "temporal_graph/metadata/%s",
}
}
// PersistTemporalNode persists a temporal node to storage // PersistTemporalNode persists a temporal node to storage
func (pm *persistenceManagerImpl) PersistTemporalNode(ctx context.Context, node *TemporalNode) error { func (pm *persistenceManagerImpl) PersistTemporalNode(ctx context.Context, node *TemporalNode) error {
pm.mu.Lock() pm.mu.Lock()
@@ -355,7 +422,7 @@ func (pm *persistenceManagerImpl) flushWriteBuffer() error {
for i, node := range pm.writeBuffer { for i, node := range pm.writeBuffer {
batch.Contexts[i] = &storage.ContextStoreItem{ batch.Contexts[i] = &storage.ContextStoreItem{
Context: node, Context: node.Context,
Roles: pm.config.EncryptionRoles, Roles: pm.config.EncryptionRoles,
} }
} }
@@ -419,8 +486,13 @@ func (pm *persistenceManagerImpl) loadFromLocalStorage(ctx context.Context) erro
return fmt.Errorf("failed to load metadata: %w", err) return fmt.Errorf("failed to load metadata: %w", err)
} }
var metadata *GraphMetadata metadataBytes, err := json.Marshal(metadataData)
if err := json.Unmarshal(metadataData.([]byte), &metadata); err != nil { if err != nil {
return fmt.Errorf("failed to marshal metadata: %w", err)
}
var metadata GraphMetadata
if err := json.Unmarshal(metadataBytes, &metadata); err != nil {
return fmt.Errorf("failed to unmarshal metadata: %w", err) return fmt.Errorf("failed to unmarshal metadata: %w", err)
} }
@@ -431,17 +503,6 @@ func (pm *persistenceManagerImpl) loadFromLocalStorage(ctx context.Context) erro
return fmt.Errorf("failed to list nodes: %w", err) return fmt.Errorf("failed to list nodes: %w", err)
} }
// Load nodes in batches
batchReq := &storage.BatchRetrieveRequest{
Keys: nodeKeys,
}
batchResult, err := pm.contextStore.BatchRetrieve(ctx, batchReq)
if err != nil {
return fmt.Errorf("failed to batch retrieve nodes: %w", err)
}
// Reconstruct graph
pm.graph.mu.Lock() pm.graph.mu.Lock()
defer pm.graph.mu.Unlock() defer pm.graph.mu.Unlock()
@@ -450,17 +511,23 @@ func (pm *persistenceManagerImpl) loadFromLocalStorage(ctx context.Context) erro
pm.graph.influences = make(map[string][]string) pm.graph.influences = make(map[string][]string)
pm.graph.influencedBy = make(map[string][]string) pm.graph.influencedBy = make(map[string][]string)
for key, result := range batchResult.Results { for _, key := range nodeKeys {
if result.Error != nil { data, err := pm.localStorage.Retrieve(ctx, key)
continue // Skip failed retrievals if err != nil {
continue
} }
var node *TemporalNode nodeBytes, err := json.Marshal(data)
if err := json.Unmarshal(result.Data.([]byte), &node); err != nil { if err != nil {
continue // Skip invalid nodes continue
} }
pm.reconstructGraphNode(node) var node TemporalNode
if err := json.Unmarshal(nodeBytes, &node); err != nil {
continue
}
pm.reconstructGraphNode(&node)
} }
return nil return nil
@@ -695,7 +762,7 @@ func (pm *persistenceManagerImpl) identifyConflicts(local, remote *GraphSnapshot
if remoteNode, exists := remote.Nodes[nodeID]; exists { if remoteNode, exists := remote.Nodes[nodeID]; exists {
if pm.hasNodeConflict(localNode, remoteNode) { if pm.hasNodeConflict(localNode, remoteNode) {
conflict := &SyncConflict{ conflict := &SyncConflict{
Type: ConflictTypeNodeMismatch, Type: ConflictVersionMismatch,
NodeID: nodeID, NodeID: nodeID,
LocalData: localNode, LocalData: localNode,
RemoteData: remoteNode, RemoteData: remoteNode,
@@ -725,15 +792,18 @@ func (pm *persistenceManagerImpl) resolveConflict(ctx context.Context, conflict
return &ConflictResolution{ return &ConflictResolution{
ConflictID: conflict.NodeID, ConflictID: conflict.NodeID,
Resolution: "merged", ResolutionMethod: "merged",
ResolvedData: resolvedNode,
ResolvedAt: time.Now(), ResolvedAt: time.Now(),
ResolvedBy: "persistence_manager",
ResultingNode: resolvedNode,
Confidence: 1.0,
Changes: []string{"merged local and remote node"},
}, nil }, nil
} }
func (pm *persistenceManagerImpl) applyConflictResolution(ctx context.Context, resolution *ConflictResolution) error { func (pm *persistenceManagerImpl) applyConflictResolution(ctx context.Context, resolution *ConflictResolution) error {
// Apply the resolved node back to the graph // Apply the resolved node back to the graph
resolvedNode := resolution.ResolvedData.(*TemporalNode) resolvedNode := resolution.ResultingNode
pm.graph.mu.Lock() pm.graph.mu.Lock()
pm.graph.nodes[resolvedNode.ID] = resolvedNode pm.graph.nodes[resolvedNode.ID] = resolvedNode

View File

@@ -3,8 +3,8 @@ package temporal
import ( import (
"context" "context"
"fmt" "fmt"
"math"
"sort" "sort"
"strings"
"sync" "sync"
"time" "time"

View File

@@ -0,0 +1,106 @@
//go:build !slurp_full
// +build !slurp_full
package temporal
import (
"context"
"fmt"
"testing"
)
func TestTemporalGraphStubBasicLifecycle(t *testing.T) {
storage := newMockStorage()
graph := NewTemporalGraph(storage)
ctx := context.Background()
address := createTestAddress("stub/basic")
contextNode := createTestContext("stub/basic", []string{"go"})
node, err := graph.CreateInitialContext(ctx, address, contextNode, "tester")
if err != nil {
t.Fatalf("expected initial context creation to succeed, got error: %v", err)
}
if node == nil {
t.Fatal("expected non-nil temporal node for initial context")
}
decision := createTestDecision("stub-dec-001", "tester", "initial evolution", ImpactLocal)
evolved, err := graph.EvolveContext(ctx, address, createTestContext("stub/basic", []string{"go", "feature"}), ReasonCodeChange, decision)
if err != nil {
t.Fatalf("expected context evolution to succeed, got error: %v", err)
}
if evolved.Version != node.Version+1 {
t.Fatalf("expected version to increment, got %d after %d", evolved.Version, node.Version)
}
latest, err := graph.GetLatestVersion(ctx, address)
if err != nil {
t.Fatalf("expected latest version retrieval to succeed, got error: %v", err)
}
if latest.Version != evolved.Version {
t.Fatalf("expected latest version %d, got %d", evolved.Version, latest.Version)
}
}
func TestTemporalInfluenceAnalyzerStub(t *testing.T) {
storage := newMockStorage()
graph := NewTemporalGraph(storage).(*temporalGraphImpl)
analyzer := NewInfluenceAnalyzer(graph)
ctx := context.Background()
addrA := createTestAddress("stub/serviceA")
addrB := createTestAddress("stub/serviceB")
if _, err := graph.CreateInitialContext(ctx, addrA, createTestContext("stub/serviceA", []string{"go"}), "tester"); err != nil {
t.Fatalf("failed to create context A: %v", err)
}
if _, err := graph.CreateInitialContext(ctx, addrB, createTestContext("stub/serviceB", []string{"go"}), "tester"); err != nil {
t.Fatalf("failed to create context B: %v", err)
}
if err := graph.AddInfluenceRelationship(ctx, addrA, addrB); err != nil {
t.Fatalf("expected influence relationship to succeed, got error: %v", err)
}
analysis, err := analyzer.AnalyzeInfluenceNetwork(ctx)
if err != nil {
t.Fatalf("expected influence analysis to succeed, got error: %v", err)
}
if analysis.TotalNodes == 0 {
t.Fatal("expected influence analysis to report at least one node")
}
}
func TestTemporalDecisionNavigatorStub(t *testing.T) {
storage := newMockStorage()
graph := NewTemporalGraph(storage).(*temporalGraphImpl)
navigator := NewDecisionNavigator(graph)
ctx := context.Background()
address := createTestAddress("stub/navigator")
if _, err := graph.CreateInitialContext(ctx, address, createTestContext("stub/navigator", []string{"go"}), "tester"); err != nil {
t.Fatalf("failed to create initial context: %v", err)
}
for i := 2; i <= 3; i++ {
id := fmt.Sprintf("stub-hop-%03d", i)
decision := createTestDecision(id, "tester", "hop", ImpactLocal)
if _, err := graph.EvolveContext(ctx, address, createTestContext("stub/navigator", []string{"go", "v"}), ReasonCodeChange, decision); err != nil {
t.Fatalf("failed to evolve context to version %d: %v", i, err)
}
}
timeline, err := navigator.GetDecisionTimeline(ctx, address, false, 0)
if err != nil {
t.Fatalf("expected timeline retrieval to succeed, got error: %v", err)
}
if timeline == nil || timeline.TotalDecisions == 0 {
t.Fatal("expected non-empty decision timeline")
}
}

View File

@@ -0,0 +1,132 @@
package temporal
import (
"context"
"fmt"
"time"
slurpContext "chorus/pkg/slurp/context"
"chorus/pkg/slurp/storage"
"chorus/pkg/ucxl"
)
// mockStorage provides an in-memory implementation of the storage interfaces used by temporal tests.
type mockStorage struct {
data map[string]interface{}
}
func newMockStorage() *mockStorage {
return &mockStorage{
data: make(map[string]interface{}),
}
}
func (ms *mockStorage) StoreContext(ctx context.Context, node *slurpContext.ContextNode, roles []string) error {
ms.data[node.UCXLAddress.String()] = node
return nil
}
func (ms *mockStorage) RetrieveContext(ctx context.Context, address ucxl.Address, role string) (*slurpContext.ContextNode, error) {
if data, exists := ms.data[address.String()]; exists {
return data.(*slurpContext.ContextNode), nil
}
return nil, storage.ErrNotFound
}
func (ms *mockStorage) UpdateContext(ctx context.Context, node *slurpContext.ContextNode, roles []string) error {
ms.data[node.UCXLAddress.String()] = node
return nil
}
func (ms *mockStorage) DeleteContext(ctx context.Context, address ucxl.Address) error {
delete(ms.data, address.String())
return nil
}
func (ms *mockStorage) ExistsContext(ctx context.Context, address ucxl.Address) (bool, error) {
_, exists := ms.data[address.String()]
return exists, nil
}
func (ms *mockStorage) ListContexts(ctx context.Context, criteria *storage.ListCriteria) ([]*slurpContext.ContextNode, error) {
results := make([]*slurpContext.ContextNode, 0)
for _, data := range ms.data {
if node, ok := data.(*slurpContext.ContextNode); ok {
results = append(results, node)
}
}
return results, nil
}
func (ms *mockStorage) SearchContexts(ctx context.Context, query *storage.SearchQuery) (*storage.SearchResults, error) {
return &storage.SearchResults{}, nil
}
func (ms *mockStorage) BatchStore(ctx context.Context, batch *storage.BatchStoreRequest) (*storage.BatchStoreResult, error) {
return &storage.BatchStoreResult{}, nil
}
func (ms *mockStorage) BatchRetrieve(ctx context.Context, batch *storage.BatchRetrieveRequest) (*storage.BatchRetrieveResult, error) {
return &storage.BatchRetrieveResult{}, nil
}
func (ms *mockStorage) GetStorageStats(ctx context.Context) (*storage.StorageStatistics, error) {
return &storage.StorageStatistics{}, nil
}
func (ms *mockStorage) Sync(ctx context.Context) error {
return nil
}
func (ms *mockStorage) Backup(ctx context.Context, destination string) error {
return nil
}
func (ms *mockStorage) Restore(ctx context.Context, source string) error {
return nil
}
// createTestAddress constructs a deterministic UCXL address for test scenarios.
func createTestAddress(path string) ucxl.Address {
return ucxl.Address{
Agent: "test-agent",
Role: "tester",
Project: "test-project",
Task: "unit-test",
TemporalSegment: ucxl.TemporalSegment{
Type: ucxl.TemporalLatest,
},
Path: path,
Raw: fmt.Sprintf("ucxl://test-agent:tester@test-project:unit-test/*^/%s", path),
}
}
// createTestContext prepares a lightweight context node for graph operations.
func createTestContext(path string, technologies []string) *slurpContext.ContextNode {
return &slurpContext.ContextNode{
Path: path,
UCXLAddress: createTestAddress(path),
Summary: fmt.Sprintf("Test context for %s", path),
Purpose: fmt.Sprintf("Test purpose for %s", path),
Technologies: technologies,
Tags: []string{"test"},
Insights: []string{"test insight"},
GeneratedAt: time.Now(),
RAGConfidence: 0.8,
}
}
// createTestDecision fabricates decision metadata to drive evolution in tests.
func createTestDecision(id, maker, rationale string, scope ImpactScope) *DecisionMetadata {
return &DecisionMetadata{
ID: id,
Maker: maker,
Rationale: rationale,
Scope: scope,
ConfidenceLevel: 0.8,
ExternalRefs: []string{},
CreatedAt: time.Now(),
ImplementationStatus: "complete",
Metadata: make(map[string]interface{}),
}
}