# BACKBEAT Schema Evolution and Versioning

This document defines how BACKBEAT message schemas evolve over time while maintaining compatibility across the CHORUS 2.0.0 ecosystem.

## Versioning Strategy

### Semantic Versioning for Schemas

BACKBEAT schemas follow semantic versioning (SemVer) with CHORUS-specific interpretations:

- **MAJOR** (`X.0.0`): Breaking changes that require code updates
- **MINOR** (`X.Y.0`): Backward-compatible additions (new optional fields, enum values)
- **PATCH** (`X.Y.Z`): Documentation updates, constraint clarifications, examples

### Schema Identification

Each schema includes version information:

```json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://chorus.services/schemas/backbeat/beatframe/v1.2.0",
  "title": "BACKBEAT BeatFrame (INT-A)",
  "version": "1.2.0"
}
```

### Message Type Versioning

Message types embed version information:

- `backbeat.beatframe.v1` → Schema version 1.x.x
- `backbeat.beatframe.v2` → Schema version 2.x.x

Only **major** version changes require new message type identifiers.

## Compatibility Matrix

### Current Schema Versions

| Interface | Schema Version | Message Type | Status |
|-----------|----------------|--------------|--------|
| INT-A (BeatFrame) | 1.0.0 | `backbeat.beatframe.v1` | Active |
| INT-B (StatusClaim) | 1.0.0 | `backbeat.statusclaim.v1` | Active |
| INT-C (BarReport) | 1.0.0 | `backbeat.barreport.v1` | Active |

### Version Compatibility Rules

1. **Minor/Patch Updates**: All v1.x.x schemas are compatible with `backbeat.*.v1` messages
2. **Major Updates**: Require new message type (e.g., `backbeat.beatframe.v2`)
3. **Transition Period**: Both old and new versions supported during migration
4. **Deprecation**: 6-month notice before removing support for old major versions

## Change Categories

### Minor Version Changes (Backward Compatible)

These changes increment the minor version (1.0.0 → 1.1.0):

#### 1. Adding Optional Fields

```json
// Before (v1.0.0)
{
  "required": ["type", "cluster_id", "beat_index"],
  "properties": {
    "type": {...},
    "cluster_id": {...},
    "beat_index": {...}
  }
}

// After (v1.1.0) - adds optional field
{
  "required": ["type", "cluster_id", "beat_index"],
  "properties": {
    "type": {...},
    "cluster_id": {...},
    "beat_index": {...},
    "priority": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10,
      "description": "Optional processing priority (1=low, 10=high)"
    }
  }
}
```

#### 2. Adding Enum Values

```json
// Before (v1.0.0)
{
  "properties": {
    "phase": {
      "enum": ["plan", "execute", "review"]
    }
  }
}

// After (v1.1.0) - adds new phase
{
  "properties": {
    "phase": {
      "enum": ["plan", "execute", "review", "cleanup"]
    }
  }
}
```

#### 3. Relaxing Constraints

```json
// Before (v1.0.0)
{
  "properties": {
    "notes": {
      "type": "string",
      "maxLength": 256
    }
  }
}

// After (v1.1.0) - allows longer notes
{
  "properties": {
    "notes": {
      "type": "string",
      "maxLength": 512
    }
  }
}
```

#### 4. Adding Properties to Objects

```json
// Before (v1.0.0)
{
  "properties": {
    "metadata": {
      "type": "object",
      "properties": {
        "version": {"type": "string"}
      }
    }
  }
}

// After (v1.1.0) - adds new metadata field
{
  "properties": {
    "metadata": {
      "type": "object",
      "properties": {
        "version": {"type": "string"},
        "source": {"type": "string"}
      }
    }
  }
}
```

### Major Version Changes (Breaking)

These changes increment the major version (1.x.x → 2.0.0):

#### 1. Removing Required Fields

```json
// v1.x.x
{
  "required": ["type", "cluster_id", "beat_index", "deprecated_field"]
}

// v2.0.0
{
  "required": ["type", "cluster_id", "beat_index"]
}
```

#### 2. Changing Field Types

```json
// v1.x.x
{
  "properties": {
    "beat_index": {"type": "integer"}
  }
}

// v2.0.0
{
  "properties": {
    "beat_index": {"type": "string"}
  }
}
```

#### 3. Removing Enum Values

```json
// v1.x.x
{
  "properties": {
    "state": {
      "enum": ["idle", "executing", "deprecated_state"]
    }
  }
}

// v2.0.0
{
  "properties": {
    "state": {
      "enum": ["idle", "executing"]
    }
  }
}
```

#### 4. Tightening Constraints

```json
// v1.x.x
{
  "properties": {
    "agent_id": {
      "type": "string",
      "maxLength": 256
    }
  }
}

// v2.0.0
{
  "properties": {
    "agent_id": {
      "type": "string",
      "maxLength": 128
    }
  }
}
```

### Patch Version Changes (Non-Breaking)

These changes increment the patch version (1.0.0 → 1.0.1):

1. **Documentation updates**
2. **Example additions**
3. **Description clarifications**
4. **Comment additions**

## Migration Strategies

### Minor Version Migration

Services automatically benefit from minor version updates:

```go
// This code works with both v1.0.0 and v1.1.0
func handleBeatFrame(frame BeatFrame) {
    // Core fields always present
    log.Printf("Beat %d in phase %s", frame.BeatIndex, frame.Phase)
    
    // New optional fields checked safely
    if frame.Priority != nil {
        log.Printf("Priority: %d", *frame.Priority)
    }
}
```

### Major Version Migration

Requires explicit handling of both versions during transition:

```go
func handleMessage(messageBytes []byte) error {
    var msgType struct {
        Type string `json:"type"`
    }
    
    if err := json.Unmarshal(messageBytes, &msgType); err != nil {
        return err
    }
    
    switch msgType.Type {
    case "backbeat.beatframe.v1":
        return handleBeatFrameV1(messageBytes)
    case "backbeat.beatframe.v2":
        return handleBeatFrameV2(messageBytes)
    default:
        return fmt.Errorf("unsupported message type: %s", msgType.Type)
    }
}
```

### Gradual Migration Process

1. **Preparation Phase** (Months 1-2)
   - Announce upcoming major version change
   - Publish v2.0.0 schemas alongside v1.x.x
   - Update documentation and examples
   - Provide migration tools and guides

2. **Dual Support Phase** (Months 3-4)
   - Services support both v1 and v2 message types
   - New services prefer v2 messages
   - Monitoring tracks v1 vs v2 usage

3. **Migration Phase** (Months 5-6)
   - All services updated to send v2 messages
   - Services still accept v1 for backward compatibility
   - Warnings logged for v1 message reception

4. **Cleanup Phase** (Month 7+)
   - Drop support for v1 messages
   - Remove v1 handling code
   - Update schemas to mark v1 as deprecated

## Implementation Guidelines

### Schema Development

1. **Start Conservative**: Begin with strict constraints, relax later if needed
2. **Plan for Growth**: Design extensible structures with optional metadata objects
3. **Document Thoroughly**: Include clear descriptions and examples
4. **Test Extensively**: Validate with real-world data before releasing

### Version Detection

Services should detect schema versions:

```go
type SchemaInfo struct {
    Version      string `json:"version"`
    MessageType  string `json:"message_type"`
    IsSupported  bool   `json:"is_supported"`
}

func detectSchemaVersion(messageType string) SchemaInfo {
    switch messageType {
    case "backbeat.beatframe.v1":
        return SchemaInfo{
            Version:     "1.x.x",
            MessageType: messageType,
            IsSupported: true,
        }
    case "backbeat.beatframe.v2":
        return SchemaInfo{
            Version:     "2.x.x", 
            MessageType: messageType,
            IsSupported: true,
        }
    default:
        return SchemaInfo{
            MessageType: messageType,
            IsSupported: false,
        }
    }
}
```

### Validation Strategy

```go
func validateWithVersionFallback(messageBytes []byte) error {
    // Try latest version first
    if err := validateV2(messageBytes); err == nil {
        return nil
    }
    
    // Fall back to previous version
    if err := validateV1(messageBytes); err == nil {
        log.Warn("Received v1 message, consider upgrading sender")
        return nil
    }
    
    return fmt.Errorf("message does not match any supported schema version")
}
```

## Testing Schema Evolution

### Compatibility Tests

```go
func TestSchemaBackwardCompatibility(t *testing.T) {
    // Test that v1.1.0 accepts all valid v1.0.0 messages
    v100Messages := loadTestMessages("v1.0.0")
    v110Schema := loadSchema("beatframe-v1.1.0.schema.json")
    
    for _, msg := range v100Messages {
        err := validateAgainstSchema(msg, v110Schema)
        assert.NoError(t, err, "v1.1.0 should accept v1.0.0 messages")
    }
}

func TestSchemaForwardCompatibility(t *testing.T) {
    // Test that v1.0.0 code gracefully handles v1.1.0 messages
    v110Message := loadTestMessage("beatframe-v1.1.0-with-new-fields.json")
    
    var beatFrame BeatFrameV1
    err := json.Unmarshal(v110Message, &beatFrame)
    assert.NoError(t, err, "v1.0.0 struct should parse v1.1.0 messages")
    
    // Core fields should be populated
    assert.NotEmpty(t, beatFrame.Type)
    assert.NotEmpty(t, beatFrame.ClusterID)
}
```

### Migration Tests

```go
func TestDualVersionSupport(t *testing.T) {
    handler := NewMessageHandler()
    
    v1Message := generateBeatFrameV1()
    v2Message := generateBeatFrameV2()
    
    // Both versions should be handled correctly
    err1 := handler.HandleMessage(v1Message)
    err2 := handler.HandleMessage(v2Message)
    
    assert.NoError(t, err1)
    assert.NoError(t, err2)
}
```

## Deprecation Process

### Marking Deprecated Features

```json
{
  "properties": {
    "legacy_field": {
      "type": "string",
      "description": "DEPRECATED: Use new_field instead. Will be removed in v2.0.0",
      "deprecated": true
    },
    "new_field": {
      "type": "string", 
      "description": "Replacement for legacy_field"
    }
  }
}
```

### Communication Timeline

1. **6 months before**: Announce deprecation in release notes
2. **3 months before**: Add deprecation warnings to schemas
3. **1 month before**: Final migration reminder
4. **Release day**: Remove deprecated features

### Tooling Support

```bash
# Check for deprecated schema usage
backbeat-validate --schemas ./schemas --dir messages/ --check-deprecated

# Migration helper
backbeat-migrate --from v1 --to v2 --dir messages/
```

## Best Practices

### For Schema Authors

1. **Communicate Early**: Announce changes well in advance
2. **Provide Tools**: Create migration utilities and documentation
3. **Monitor Usage**: Track which versions are being used
4. **Be Conservative**: Prefer minor over major version changes

### For Service Developers

1. **Stay Updated**: Subscribe to schema change notifications
2. **Plan for Migration**: Build version handling into your services
3. **Test Thoroughly**: Validate against multiple schema versions
4. **Monitor Compatibility**: Alert on unsupported message versions

### For Operations Teams

1. **Version Tracking**: Monitor which schema versions are active
2. **Migration Planning**: Coordinate major version migrations
3. **Rollback Capability**: Be prepared to revert if migrations fail
4. **Performance Impact**: Monitor schema validation performance

## Future Considerations

### Planned Enhancements

1. **Schema Registry**: Centralized schema version management
2. **Auto-Migration**: Tools to automatically update message formats
3. **Version Negotiation**: Services negotiate supported versions
4. **Schema Analytics**: Usage metrics and compatibility reporting

### Long-term Vision

- **Continuous Evolution**: Schemas evolve without breaking existing services
- **Zero-Downtime Updates**: Schema changes deploy without service interruption  
- **Automated Testing**: CI/CD pipelines validate schema compatibility
- **Self-Healing**: Services automatically adapt to schema changes