tony/bzzz
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ee6bb095114bb603e0be9d583383cc6b053a574d
bzzz/old-docs/TECHNICAL_ARCHITECTURE.md
anthonyrawlins ee6bb09511 Complete Phase 2B documentation suite and implementation 
🎉 MAJOR MILESTONE: Complete BZZZ Phase 2B documentation and core implementation

## Documentation Suite (7,000+ lines)
- ✅ User Manual: Comprehensive guide with practical examples
- ✅ API Reference: Complete REST API documentation
- ✅ SDK Documentation: Multi-language SDK guide (Go, Python, JS, Rust)
- ✅ Developer Guide: Development setup and contribution procedures
- ✅ Architecture Documentation: Detailed system design with ASCII diagrams
- ✅ Technical Report: Performance analysis and benchmarks
- ✅ Security Documentation: Comprehensive security model
- ✅ Operations Guide: Production deployment and monitoring
- ✅ Documentation Index: Cross-referenced navigation system

## SDK Examples & Integration
- 🔧 Go SDK: Simple client, event streaming, crypto operations
- 🐍 Python SDK: Async client with comprehensive examples
- 📜 JavaScript SDK: Collaborative agent implementation
- 🦀 Rust SDK: High-performance monitoring system
- 📖 Multi-language README with setup instructions

## Core Implementation
- 🔐 Age encryption implementation (pkg/crypto/age_crypto.go)
- 🗂️ Shamir secret sharing (pkg/crypto/shamir.go)
- 💾 DHT encrypted storage (pkg/dht/encrypted_storage.go)
- 📤 UCXL decision publisher (pkg/ucxl/decision_publisher.go)
- 🔄 Updated main.go with Phase 2B integration

## Project Organization
- 📂 Moved legacy docs to old-docs/ directory
- 🎯 Comprehensive README.md update with modern structure
- 🔗 Full cross-reference system between all documentation
- 📊 Production-ready deployment procedures

## Quality Assurance
- ✅ All documentation cross-referenced and validated
- ✅ Working code examples in multiple languages
- ✅ Production deployment procedures tested
- ✅ Security best practices implemented
- ✅ Performance benchmarks documented

Ready for production deployment and community adoption.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-08 19:57:40 +10:00

567 lines
24 KiB
Markdown
Raw Blame History

# BZZZ v2 Technical Architecture: UCXL/UCXI Integration
## 1. Architecture Overview
BZZZ v2 transforms from a GitHub Issues-based task coordination system to a semantic context publishing platform built on the Universal Context eXchange Language (UCXL) protocol. The system maintains its distributed P2P foundation while adding sophisticated temporal navigation, decision graph publishing, and integration with the broader CHORUS infrastructure.
```
┌─────────────────────────────────────────────────────────┐
│ UCXL Ecosystem │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ UCXL │ │ UCXL │ │
│ │ Validator │ │ Browser │ │
│ │ (Online) │ │ (Time Machine) │ │
│ └─────────────────┘ └─────────────────┘ │
└─────────────────────────────────────────────────────────┘
│
┌─────────────────────────────────────────────────────────┐
│ BZZZ v2 Core │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ UCXI │ │ Decision │ │
│ │ Interface │────│ Publishing │ │
│ │ Server │ │ Pipeline │ │
│ └─────────────────┘ └─────────────────┘ │
│ │ │ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Temporal │ │ Context │ │
│ │ Navigation │────│ Storage │ │
│ │ Engine │ │ Backend │ │
│ └─────────────────┘ └─────────────────┘ │
│ │ │ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ UCXL │ │ P2P DHT │ │
│ │ Address │────│ Resolution │ │
│ │ Parser │ │ Network │ │
│ └─────────────────┘ └─────────────────┘ │
└─────────────────────────────────────────────────────────┘
│
┌─────────────────────────────────────────────────────────┐
│ CHORUS Infrastructure │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ SLURP │ │ WHOOSH │ │
│ │ Context │────│ Search │ │
│ │ Ingestion │ │ Indexing │ │
│ └─────────────────┘ └─────────────────┘ │
│ │ │ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ N8N │ │ GitLab │ │
│ │ Automation │────│ Integration │ │
│ │ Workflows │ │ (Optional) │ │
│ └─────────────────┘ └─────────────────┘ │
└─────────────────────────────────────────────────────────┘
```
## 2. Core Components
### 2.1 UCXL Address Parser (`pkg/protocol/ucxl_address.go`)
Replaces the existing `pkg/protocol/uri.go` with full UCXL protocol support.
```go
type UCXLAddress struct {
// Core addressing components
Agent string `json:"agent"` // e.g., "gpt4", "claude", "any"
Role string `json:"role"` // e.g., "architect", "reviewer", "any"
Project string `json:"project"` // e.g., "bzzz", "chorus", "any"
Task string `json:"task"` // e.g., "v2-migration", "auth", "any"
// Temporal navigation
TemporalSegment string `json:"temporal_segment"` // "~~", "^^", "*^", "*~", ISO8601
// Resource path
Path string `json:"path"` // "/decisions/architecture.json"
// Standard URI components
Query string `json:"query,omitempty"`
Fragment string `json:"fragment,omitempty"`
Raw string `json:"raw"`
}
// Navigation tokens
const (
TemporalBackward = "~~" // Navigate backward in time
TemporalForward = "^^" // Navigate forward in time
TemporalLatest = "*^" // Latest entry
TemporalFirst = "*~" // First entry
)
```
#### Key Methods:
- `ParseUCXLAddress(uri string) (*UCXLAddress, error)`
- `Normalize()` - Standardize address format
- `Matches(other *UCXLAddress) bool` - Wildcard matching with `any:any`
- `GetTemporalTarget() (time.Time, error)` - Resolve temporal navigation
- `ToStorageKey() string` - Generate storage backend key
### 2.2 UCXI Interface Server (`pkg/ucxi/server.go`)
HTTP server implementing UCXI operations with REST-like semantics.
```go
type UCXIServer struct {
contextStore storage.ContextStore
temporalIndex temporal.Index
p2pNode *p2p.Node
resolver *routing.SemanticRouter
}
// UCXI Operations
type UCXIOperations interface {
GET(address *UCXLAddress) (*ContextEntry, error)
PUT(address *UCXLAddress, content interface{}) error
POST(address *UCXLAddress, content interface{}) (*UCXLAddress, error)
DELETE(address *UCXLAddress) error
ANNOUNCE(address *UCXLAddress, metadata ContextMetadata) error
// Extended operations
NAVIGATE(address *UCXLAddress, direction string) (*UCXLAddress, error)
QUERY(pattern *UCXLAddress) ([]*ContextEntry, error)
SUBSCRIBE(pattern *UCXLAddress, callback func(*ContextEntry)) error
}
```
#### HTTP Endpoints:
- `GET /ucxi/{agent}:{role}@{project}:{task}/{temporal}/{path}`
- `PUT /ucxi/{agent}:{role}@{project}:{task}/{temporal}/{path}`
- `POST /ucxi/{agent}:{role}@{project}:{task}/{temporal}/`
- `DELETE /ucxi/{agent}:{role}@{project}:{task}/{temporal}/{path}`
- `POST /ucxi/announce`
- `GET /ucxi/navigate/{direction}`
- `GET /ucxi/query?pattern={pattern}`
- `POST /ucxi/subscribe`
### 2.3 Temporal Navigation Engine (`pkg/temporal/navigator.go`)
Handles time-based context navigation and maintains temporal consistency.
```go
type TemporalNavigator struct {
index TemporalIndex
snapshots SnapshotManager
store storage.ContextStore
}
type TemporalIndex struct {
// Address pattern -> sorted temporal entries
patterns map[string][]TemporalEntry
mutex sync.RWMutex
}
type TemporalEntry struct {
Timestamp time.Time `json:"timestamp"`
Version int64 `json:"version"`
Address UCXLAddress `json:"address"`
Checksum string `json:"checksum"`
}
// Navigation methods
func (tn *TemporalNavigator) NavigateBackward(address *UCXLAddress) (*UCXLAddress, error)
func (tn *TemporalNavigator) NavigateForward(address *UCXLAddress) (*UCXLAddress, error)
func (tn *TemporalNavigator) GetLatest(address *UCXLAddress) (*UCXLAddress, error)
func (tn *TemporalNavigator) GetFirst(address *UCXLAddress) (*UCXLAddress, error)
func (tn *TemporalNavigator) GetAtTime(address *UCXLAddress, timestamp time.Time) (*UCXLAddress, error)
```
### 2.4 Context Storage Backend (`pkg/storage/context_store.go`)
Versioned storage system supporting both local and distributed storage.
```go
type ContextStore interface {
Store(address *UCXLAddress, entry *ContextEntry) error
Retrieve(address *UCXLAddress) (*ContextEntry, error)
Delete(address *UCXLAddress) error
List(pattern *UCXLAddress) ([]*ContextEntry, error)
// Versioning
GetVersion(address *UCXLAddress, version int64) (*ContextEntry, error)
ListVersions(address *UCXLAddress) ([]VersionInfo, error)
// Temporal operations
GetAtTime(address *UCXLAddress, timestamp time.Time) (*ContextEntry, error)
GetRange(address *UCXLAddress, start, end time.Time) ([]*ContextEntry, error)
}
type ContextEntry struct {
Address UCXLAddress `json:"address"`
Content map[string]interface{} `json:"content"`
Metadata ContextMetadata `json:"metadata"`
Version int64 `json:"version"`
Checksum string `json:"checksum"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
```
#### Storage Backends:
- **LocalFS**: File-based storage for development
- **BadgerDB**: Embedded key-value store for production
- **NFS**: Distributed storage across CHORUS cluster
- **IPFS**: Content-addressed storage (future)
### 2.5 P2P DHT Resolution (`pkg/dht/ucxl_resolver.go`)
Extends existing libp2p DHT for UCXL address resolution and discovery.
```go
type UCXLResolver struct {
dht *dht.IpfsDHT
localStore storage.ContextStore
peerCache map[peer.ID]*PeerCapabilities
router *routing.SemanticRouter
}
type PeerCapabilities struct {
SupportedAgents []string `json:"supported_agents"`
SupportedRoles []string `json:"supported_roles"`
SupportedProjects []string `json:"supported_projects"`
LastSeen time.Time `json:"last_seen"`
}
// Resolution methods
func (ur *UCXLResolver) Resolve(address *UCXLAddress) ([]*ContextEntry, error)
func (ur *UCXLResolver) Announce(address *UCXLAddress, metadata ContextMetadata) error
func (ur *UCXLResolver) FindProviders(address *UCXLAddress) ([]peer.ID, error)
func (ur *UCXLResolver) Subscribe(pattern *UCXLAddress) (<-chan *ContextEntry, error)
```
#### DHT Operations:
- **Provider Records**: Map UCXL addresses to providing peers
- **Capability Announcements**: Broadcast agent/role/project support
- **Semantic Routing**: Route `any:any` patterns to appropriate peers
- **Context Discovery**: Find contexts matching wildcard patterns
### 2.6 Decision Publishing Pipeline (`pkg/decisions/publisher.go`)
Publishes structured decision nodes to SLURP after agent task completion.
```go
type DecisionPublisher struct {
slurpClient *integration.SLURPClient
validator *validation.CitationValidator
curator *curation.DecisionCurator
contextStore storage.ContextStore
}
type DecisionNode struct {
DecisionID string `json:"decision_id"`
UCXLAddress string `json:"ucxl_address"`
Timestamp time.Time `json:"timestamp"`
AgentID string `json:"agent_id"`
DecisionType string `json:"decision_type"`
Context DecisionContext `json:"context"`
Justification Justification `json:"justification"`
Citations []Citation `json:"citations"`
Impacts []Impact `json:"impacts"`
}
type Justification struct {
Reasoning string `json:"reasoning"`
AlternativesConsidered []string `json:"alternatives_considered"`
Criteria []string `json:"criteria"`
Confidence float64 `json:"confidence"`
}
type Citation struct {
Type string `json:"type"` // "justified_by", "references", "contradicts"
UCXLAddress string `json:"ucxl_address"`
Relevance string `json:"relevance"` // "high", "medium", "low"
Excerpt string `json:"excerpt"`
Strength float64 `json:"strength"`
}
```
## 3. Integration Points
### 3.1 SLURP Context Ingestion
Decision nodes are published to SLURP for global context graph building:
```go
type SLURPClient struct {
baseURL string
httpClient *http.Client
apiKey string
}
func (sc *SLURPClient) PublishDecision(node *DecisionNode) error
func (sc *SLURPClient) QueryContext(query string) ([]*ContextEntry, error)
func (sc *SLURPClient) GetJustificationChain(decisionID string) ([]*DecisionNode, error)
```
**SLURP Integration Flow:**
1. Agent completes task (execution, review, architecture)
2. Decision curator extracts decision-worthy content
3. Citation validator checks justification chains
4. Decision publisher sends structured node to SLURP
5. SLURP ingests into global context graph
### 3.2 WHOOSH Search Integration
UCXL addresses and content indexed for semantic search:
```go
// Index UCXL addresses in WHOOSH
type UCXLIndexer struct {
whooshClient *whoosh.Client
indexName string
}
func (ui *UCXLIndexer) IndexContext(entry *ContextEntry) error
func (ui *UCXLIndexer) SearchAddresses(query string) ([]*UCXLAddress, error)
func (ui *UCXLIndexer) SearchContent(pattern *UCXLAddress, query string) ([]*ContextEntry, error)
func (ui *UCXLIndexer) SearchTemporal(timeQuery string) ([]*ContextEntry, error)
```
**Search Capabilities:**
- Address pattern search (`agent:architect@*:*`)
- Temporal search (`decisions after 2025-08-01`)
- Content full-text search with UCXL scoping
- Citation graph exploration
### 3.3 Agent MCP Tools
Update MCP server with UCXI operation tools:
```typescript
// mcp-server/src/tools/ucxi-tools.ts
export const ucxiTools = {
ucxi_get: {
name: "ucxi_get",
description: "Retrieve context from UCXL address",
inputSchema: {
type: "object",
properties: {
address: { type: "string" },
temporal: { type: "string", enum: ["~~", "^^", "*^", "*~"] }
}
}
},
ucxi_put: {
name: "ucxi_put",
description: "Store context at UCXL address",
inputSchema: {
type: "object",
properties: {
address: { type: "string" },
content: { type: "object" },
metadata: { type: "object" }
}
}
},
ucxi_announce: {
name: "ucxi_announce",
description: "Announce context availability",
inputSchema: {
type: "object",
properties: {
address: { type: "string" },
capabilities: { type: "array" }
}
}
}
}
```
## 4. Data Flow Architecture
### 4.1 Context Publishing Flow
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ GPT-4 Agent │ │ Decision │ │ UCXI │
│ Completes │────│ Curation │────│ Storage │
│ Task │ │ Pipeline │ │ Backend │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Task Result │ │ Structured │ │ Versioned │
│ Analysis │────│ Decision Node │────│ Context │
│ │ │ Generation │ │ Storage │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Citation │ │ SLURP │ │ P2P DHT │
│ Validation │────│ Publishing │────│ Announcement │
│ │ │ │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
```
### 4.2 Context Resolution Flow
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Agent │ │ UCXL │ │ Temporal │
│ UCXI Request │────│ Address │────│ Navigation │
│ │ │ Parser │ │ Engine │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Local Cache │ │ Semantic │ │ Context │
│ Lookup │────│ Router │────│ Retrieval │
│ │ │ │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Cache Hit │ │ P2P DHT │ │ Context │
│ Response │────│ Resolution │────│ Response │
│ │ │ │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
```
## 5. Configuration & Deployment
### 5.1 BZZZ v2 Configuration
```yaml
# config/bzzz-v2.yaml
bzzz:
version: "2.0"
protocol: "ucxl"
ucxi:
server:
host: "0.0.0.0"
port: 8080
tls_enabled: true
cert_file: "/etc/bzzz/tls/cert.pem"
key_file: "/etc/bzzz/tls/key.pem"
storage:
backend: "badgerdb" # options: localfs, badgerdb, nfs
path: "/var/lib/bzzz/context"
max_size: "10GB"
compression: true
temporal:
retention_period: "90d"
snapshot_interval: "1h"
max_versions: 100
p2p:
listen_addrs:
- "/ip4/0.0.0.0/tcp/4001"
- "/ip6/::/tcp/4001"
bootstrap_peers: []
dht_mode: "server"
slurp:
endpoint: "http://slurp.chorus.local:8080"
api_key: "${SLURP_API_KEY}"
publish_decisions: true
batch_size: 10
agent:
id: "bzzz-${NODE_ID}"
roles: ["architect", "reviewer", "implementer"]
supported_agents: ["gpt4", "claude"]
monitoring:
metrics_port: 9090
health_port: 8081
log_level: "info"
```
### 5.2 Docker Swarm Deployment
```yaml
# infrastructure/docker-compose.swarm.yml
version: '3.8'
services:
bzzz-v2:
image: registry.home.deepblack.cloud/bzzz:v2-latest
deploy:
replicas: 3
placement:
constraints:
- node.role == worker
resources:
limits:
memory: 2GB
cpus: '1.0'
environment:
- NODE_ID={{.Task.Slot}}
- SLURP_API_KEY=${SLURP_API_KEY}
volumes:
- bzzz-context:/var/lib/bzzz/context
- /rust/containers/bzzz/config:/etc/bzzz:ro
networks:
- bzzz-net
- chorus-net
ports:
- "808{{.Task.Slot}}:8080" # UCXI server
- "400{{.Task.Slot}}:4001" # P2P libp2p
volumes:
bzzz-context:
driver: local
driver_opts:
type: nfs
o: addr=192.168.1.72,rw
device: ":/rust/containers/bzzz/data"
networks:
bzzz-net:
external: true
chorus-net:
external: true
```
## 6. Performance & Scalability
### 6.1 Performance Targets
- **Address Resolution**: < 100ms for cached contexts
- **Temporal Navigation**: < 50ms for recent contexts
- **Decision Publishing**: < 5s end-to-end to SLURP
- **Concurrent Operations**: 1000+ UCXI operations/second
- **Storage Efficiency**: 70%+ compression ratio
### 6.2 Scaling Strategy
- **Horizontal Scaling**: Add nodes to P2P network
- **Context Sharding**: Distribute context by address hash
- **Temporal Sharding**: Partition by time ranges
- **Caching Hierarchy**: Local → Cluster → P2P resolution
- **Load Balancing**: UCXI requests across cluster nodes
### 6.3 Monitoring & Observability
```go
// Prometheus metrics
var (
ucxiOperationsTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "bzzz_ucxi_operations_total",
Help: "Total number of UCXI operations",
},
[]string{"operation", "status"},
)
contextResolutionDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "bzzz_context_resolution_duration_seconds",
Help: "Time spent resolving UCXL addresses",
},
[]string{"resolution_method"},
)
decisionPublishingDuration = prometheus.NewHistogram(
prometheus.HistogramOpts{
Name: "bzzz_decision_publishing_duration_seconds",
Help: "Time spent publishing decisions to SLURP",
},
)
)
```
This technical architecture provides the foundation for implementing BZZZ v2 as a sophisticated UCXL-based semantic context publishing system while maintaining the distributed P2P characteristics that make it resilient and scalable within the CHORUS infrastructure.
Reference in New Issue View Git Blame Copy Permalink