BACKBEAT/contracts/docs/schema-evolution.md

BACKBEAT Schema Evolution and Versioning

This document defines how BACKBEAT message schemas evolve over time while maintaining compatibility across the CHORUS 2.0.0 ecosystem.

Versioning Strategy

Semantic Versioning for Schemas

BACKBEAT schemas follow semantic versioning (SemVer) with CHORUS-specific interpretations:

• MAJOR (X.0.0): Breaking changes that require code updates
• MINOR (X.Y.0): Backward-compatible additions (new optional fields, enum values)
• PATCH (X.Y.Z): Documentation updates, constraint clarifications, examples

Schema Identification

Each schema includes version information:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://chorus.services/schemas/backbeat/beatframe/v1.2.0",
  "title": "BACKBEAT BeatFrame (INT-A)",
  "version": "1.2.0"
}

Message Type Versioning

Message types embed version information:

• backbeat.beatframe.v1 → Schema version 1.x.x
• backbeat.beatframe.v2 → Schema version 2.x.x

Only major version changes require new message type identifiers.

Compatibility Matrix

Current Schema Versions

Interface	Schema Version	Message Type	Status
INT-A (BeatFrame)	1.0.0	backbeat.beatframe.v1	Active
INT-B (StatusClaim)	1.0.0	backbeat.statusclaim.v1	Active
INT-C (BarReport)	1.0.0	backbeat.barreport.v1	Active

Version Compatibility Rules

1. Minor/Patch Updates: All v1.x.x schemas are compatible with backbeat.*.v1 messages
2. Major Updates: Require new message type (e.g., backbeat.beatframe.v2)
3. Transition Period: Both old and new versions supported during migration
4. Deprecation: 6-month notice before removing support for old major versions

Change Categories

Minor Version Changes (Backward Compatible)

These changes increment the minor version (1.0.0 → 1.1.0):

1. Adding Optional Fields

// Before (v1.0.0)
{
  "required": ["type", "cluster_id", "beat_index"],
  "properties": {
    "type": {...},
    "cluster_id": {...},
    "beat_index": {...}
  }
}

// After (v1.1.0) - adds optional field
{
  "required": ["type", "cluster_id", "beat_index"],
  "properties": {
    "type": {...},
    "cluster_id": {...},
    "beat_index": {...},
    "priority": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10,
      "description": "Optional processing priority (1=low, 10=high)"
    }
  }
}

2. Adding Enum Values

// Before (v1.0.0)
{
  "properties": {
    "phase": {
      "enum": ["plan", "execute", "review"]
    }
  }
}

// After (v1.1.0) - adds new phase
{
  "properties": {
    "phase": {
      "enum": ["plan", "execute", "review", "cleanup"]
    }
  }
}

3. Relaxing Constraints

// Before (v1.0.0)
{
  "properties": {
    "notes": {
      "type": "string",
      "maxLength": 256
    }
  }
}

// After (v1.1.0) - allows longer notes
{
  "properties": {
    "notes": {
      "type": "string",
      "maxLength": 512
    }
  }
}

4. Adding Properties to Objects

// Before (v1.0.0)
{
  "properties": {
    "metadata": {
      "type": "object",
      "properties": {
        "version": {"type": "string"}
      }
    }
  }
}

// After (v1.1.0) - adds new metadata field
{
  "properties": {
    "metadata": {
      "type": "object",
      "properties": {
        "version": {"type": "string"},
        "source": {"type": "string"}
      }
    }
  }
}

Major Version Changes (Breaking)

These changes increment the major version (1.x.x → 2.0.0):

1. Removing Required Fields

// v1.x.x
{
  "required": ["type", "cluster_id", "beat_index", "deprecated_field"]
}

// v2.0.0
{
  "required": ["type", "cluster_id", "beat_index"]
}

2. Changing Field Types

// v1.x.x
{
  "properties": {
    "beat_index": {"type": "integer"}
  }
}

// v2.0.0
{
  "properties": {
    "beat_index": {"type": "string"}
  }
}

3. Removing Enum Values

// v1.x.x
{
  "properties": {
    "state": {
      "enum": ["idle", "executing", "deprecated_state"]
    }
  }
}

// v2.0.0
{
  "properties": {
    "state": {
      "enum": ["idle", "executing"]
    }
  }
}

4. Tightening Constraints

// v1.x.x
{
  "properties": {
    "agent_id": {
      "type": "string",
      "maxLength": 256
    }
  }
}

// v2.0.0
{
  "properties": {
    "agent_id": {
      "type": "string",
      "maxLength": 128
    }
  }
}

Patch Version Changes (Non-Breaking)

These changes increment the patch version (1.0.0 → 1.0.1):

1. Documentation updates
2. Example additions
3. Description clarifications
4. Comment additions

Migration Strategies

Minor Version Migration

Services automatically benefit from minor version updates:

// This code works with both v1.0.0 and v1.1.0
func handleBeatFrame(frame BeatFrame) {
    // Core fields always present
    log.Printf("Beat %d in phase %s", frame.BeatIndex, frame.Phase)
    
    // New optional fields checked safely
    if frame.Priority != nil {
        log.Printf("Priority: %d", *frame.Priority)
    }
}

Major Version Migration

Requires explicit handling of both versions during transition:

func handleMessage(messageBytes []byte) error {
    var msgType struct {
        Type string `json:"type"`
    }
    
    if err := json.Unmarshal(messageBytes, &msgType); err != nil {
        return err
    }
    
    switch msgType.Type {
    case "backbeat.beatframe.v1":
        return handleBeatFrameV1(messageBytes)
    case "backbeat.beatframe.v2":
        return handleBeatFrameV2(messageBytes)
    default:
        return fmt.Errorf("unsupported message type: %s", msgType.Type)
    }
}

Gradual Migration Process

1. Preparation Phase (Months 1-2)

   • Announce upcoming major version change
   • Publish v2.0.0 schemas alongside v1.x.x
   • Update documentation and examples
   • Provide migration tools and guides

2. Dual Support Phase (Months 3-4)

   • Services support both v1 and v2 message types
   • New services prefer v2 messages
   • Monitoring tracks v1 vs v2 usage

3. Migration Phase (Months 5-6)

   • All services updated to send v2 messages
   • Services still accept v1 for backward compatibility
   • Warnings logged for v1 message reception

4. Cleanup Phase (Month 7+)

   • Drop support for v1 messages
   • Remove v1 handling code
   • Update schemas to mark v1 as deprecated

Implementation Guidelines

Schema Development

1. Start Conservative: Begin with strict constraints, relax later if needed
2. Plan for Growth: Design extensible structures with optional metadata objects
3. Document Thoroughly: Include clear descriptions and examples
4. Test Extensively: Validate with real-world data before releasing

Version Detection

Services should detect schema versions:

type SchemaInfo struct {
    Version      string `json:"version"`
    MessageType  string `json:"message_type"`
    IsSupported  bool   `json:"is_supported"`
}

func detectSchemaVersion(messageType string) SchemaInfo {
    switch messageType {
    case "backbeat.beatframe.v1":
        return SchemaInfo{
            Version:     "1.x.x",
            MessageType: messageType,
            IsSupported: true,
        }
    case "backbeat.beatframe.v2":
        return SchemaInfo{
            Version:     "2.x.x", 
            MessageType: messageType,
            IsSupported: true,
        }
    default:
        return SchemaInfo{
            MessageType: messageType,
            IsSupported: false,
        }
    }
}

Validation Strategy

func validateWithVersionFallback(messageBytes []byte) error {
    // Try latest version first
    if err := validateV2(messageBytes); err == nil {
        return nil
    }
    
    // Fall back to previous version
    if err := validateV1(messageBytes); err == nil {
        log.Warn("Received v1 message, consider upgrading sender")
        return nil
    }
    
    return fmt.Errorf("message does not match any supported schema version")
}

Testing Schema Evolution

Compatibility Tests

func TestSchemaBackwardCompatibility(t *testing.T) {
    // Test that v1.1.0 accepts all valid v1.0.0 messages
    v100Messages := loadTestMessages("v1.0.0")
    v110Schema := loadSchema("beatframe-v1.1.0.schema.json")
    
    for _, msg := range v100Messages {
        err := validateAgainstSchema(msg, v110Schema)
        assert.NoError(t, err, "v1.1.0 should accept v1.0.0 messages")
    }
}

func TestSchemaForwardCompatibility(t *testing.T) {
    // Test that v1.0.0 code gracefully handles v1.1.0 messages
    v110Message := loadTestMessage("beatframe-v1.1.0-with-new-fields.json")
    
    var beatFrame BeatFrameV1
    err := json.Unmarshal(v110Message, &beatFrame)
    assert.NoError(t, err, "v1.0.0 struct should parse v1.1.0 messages")
    
    // Core fields should be populated
    assert.NotEmpty(t, beatFrame.Type)
    assert.NotEmpty(t, beatFrame.ClusterID)
}

Migration Tests

func TestDualVersionSupport(t *testing.T) {
    handler := NewMessageHandler()
    
    v1Message := generateBeatFrameV1()
    v2Message := generateBeatFrameV2()
    
    // Both versions should be handled correctly
    err1 := handler.HandleMessage(v1Message)
    err2 := handler.HandleMessage(v2Message)
    
    assert.NoError(t, err1)
    assert.NoError(t, err2)
}

Deprecation Process

Marking Deprecated Features

{
  "properties": {
    "legacy_field": {
      "type": "string",
      "description": "DEPRECATED: Use new_field instead. Will be removed in v2.0.0",
      "deprecated": true
    },
    "new_field": {
      "type": "string", 
      "description": "Replacement for legacy_field"
    }
  }
}

Communication Timeline

1. 6 months before: Announce deprecation in release notes
2. 3 months before: Add deprecation warnings to schemas
3. 1 month before: Final migration reminder
4. Release day: Remove deprecated features

Tooling Support

# Check for deprecated schema usage
backbeat-validate --schemas ./schemas --dir messages/ --check-deprecated

# Migration helper
backbeat-migrate --from v1 --to v2 --dir messages/

Best Practices

For Schema Authors

1. Communicate Early: Announce changes well in advance
2. Provide Tools: Create migration utilities and documentation
3. Monitor Usage: Track which versions are being used
4. Be Conservative: Prefer minor over major version changes

For Service Developers

1. Stay Updated: Subscribe to schema change notifications
2. Plan for Migration: Build version handling into your services
3. Test Thoroughly: Validate against multiple schema versions
4. Monitor Compatibility: Alert on unsupported message versions

For Operations Teams

1. Version Tracking: Monitor which schema versions are active
2. Migration Planning: Coordinate major version migrations
3. Rollback Capability: Be prepared to revert if migrations fail
4. Performance Impact: Monitor schema validation performance

Future Considerations

Planned Enhancements

1. Schema Registry: Centralized schema version management
2. Auto-Migration: Tools to automatically update message formats
3. Version Negotiation: Services negotiate supported versions
4. Schema Analytics: Usage metrics and compatibility reporting

Long-term Vision

• Continuous Evolution: Schemas evolve without breaking existing services
• Zero-Downtime Updates: Schema changes deploy without service interruption
• Automated Testing: CI/CD pipelines validate schema compatibility
• Self-Healing: Services automatically adapt to schema changes