507 lines
12 KiB
Markdown
507 lines
12 KiB
Markdown
# BACKBEAT Schema Evolution and Versioning
|
|
|
|
This document defines how BACKBEAT message schemas evolve over time while maintaining compatibility across the CHORUS 2.0.0 ecosystem.
|
|
|
|
## Versioning Strategy
|
|
|
|
### Semantic Versioning for Schemas
|
|
|
|
BACKBEAT schemas follow semantic versioning (SemVer) with CHORUS-specific interpretations:
|
|
|
|
- **MAJOR** (`X.0.0`): Breaking changes that require code updates
|
|
- **MINOR** (`X.Y.0`): Backward-compatible additions (new optional fields, enum values)
|
|
- **PATCH** (`X.Y.Z`): Documentation updates, constraint clarifications, examples
|
|
|
|
### Schema Identification
|
|
|
|
Each schema includes version information:
|
|
|
|
```json
|
|
{
|
|
"$schema": "http://json-schema.org/draft-07/schema#",
|
|
"$id": "https://chorus.services/schemas/backbeat/beatframe/v1.2.0",
|
|
"title": "BACKBEAT BeatFrame (INT-A)",
|
|
"version": "1.2.0"
|
|
}
|
|
```
|
|
|
|
### Message Type Versioning
|
|
|
|
Message types embed version information:
|
|
|
|
- `backbeat.beatframe.v1` → Schema version 1.x.x
|
|
- `backbeat.beatframe.v2` → Schema version 2.x.x
|
|
|
|
Only **major** version changes require new message type identifiers.
|
|
|
|
## Compatibility Matrix
|
|
|
|
### Current Schema Versions
|
|
|
|
| Interface | Schema Version | Message Type | Status |
|
|
|-----------|----------------|--------------|--------|
|
|
| INT-A (BeatFrame) | 1.0.0 | `backbeat.beatframe.v1` | Active |
|
|
| INT-B (StatusClaim) | 1.0.0 | `backbeat.statusclaim.v1` | Active |
|
|
| INT-C (BarReport) | 1.0.0 | `backbeat.barreport.v1` | Active |
|
|
|
|
### Version Compatibility Rules
|
|
|
|
1. **Minor/Patch Updates**: All v1.x.x schemas are compatible with `backbeat.*.v1` messages
|
|
2. **Major Updates**: Require new message type (e.g., `backbeat.beatframe.v2`)
|
|
3. **Transition Period**: Both old and new versions supported during migration
|
|
4. **Deprecation**: 6-month notice before removing support for old major versions
|
|
|
|
## Change Categories
|
|
|
|
### Minor Version Changes (Backward Compatible)
|
|
|
|
These changes increment the minor version (1.0.0 → 1.1.0):
|
|
|
|
#### 1. Adding Optional Fields
|
|
|
|
```json
|
|
// Before (v1.0.0)
|
|
{
|
|
"required": ["type", "cluster_id", "beat_index"],
|
|
"properties": {
|
|
"type": {...},
|
|
"cluster_id": {...},
|
|
"beat_index": {...}
|
|
}
|
|
}
|
|
|
|
// After (v1.1.0) - adds optional field
|
|
{
|
|
"required": ["type", "cluster_id", "beat_index"],
|
|
"properties": {
|
|
"type": {...},
|
|
"cluster_id": {...},
|
|
"beat_index": {...},
|
|
"priority": {
|
|
"type": "integer",
|
|
"minimum": 1,
|
|
"maximum": 10,
|
|
"description": "Optional processing priority (1=low, 10=high)"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 2. Adding Enum Values
|
|
|
|
```json
|
|
// Before (v1.0.0)
|
|
{
|
|
"properties": {
|
|
"phase": {
|
|
"enum": ["plan", "execute", "review"]
|
|
}
|
|
}
|
|
}
|
|
|
|
// After (v1.1.0) - adds new phase
|
|
{
|
|
"properties": {
|
|
"phase": {
|
|
"enum": ["plan", "execute", "review", "cleanup"]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 3. Relaxing Constraints
|
|
|
|
```json
|
|
// Before (v1.0.0)
|
|
{
|
|
"properties": {
|
|
"notes": {
|
|
"type": "string",
|
|
"maxLength": 256
|
|
}
|
|
}
|
|
}
|
|
|
|
// After (v1.1.0) - allows longer notes
|
|
{
|
|
"properties": {
|
|
"notes": {
|
|
"type": "string",
|
|
"maxLength": 512
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 4. Adding Properties to Objects
|
|
|
|
```json
|
|
// Before (v1.0.0)
|
|
{
|
|
"properties": {
|
|
"metadata": {
|
|
"type": "object",
|
|
"properties": {
|
|
"version": {"type": "string"}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// After (v1.1.0) - adds new metadata field
|
|
{
|
|
"properties": {
|
|
"metadata": {
|
|
"type": "object",
|
|
"properties": {
|
|
"version": {"type": "string"},
|
|
"source": {"type": "string"}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Major Version Changes (Breaking)
|
|
|
|
These changes increment the major version (1.x.x → 2.0.0):
|
|
|
|
#### 1. Removing Required Fields
|
|
|
|
```json
|
|
// v1.x.x
|
|
{
|
|
"required": ["type", "cluster_id", "beat_index", "deprecated_field"]
|
|
}
|
|
|
|
// v2.0.0
|
|
{
|
|
"required": ["type", "cluster_id", "beat_index"]
|
|
}
|
|
```
|
|
|
|
#### 2. Changing Field Types
|
|
|
|
```json
|
|
// v1.x.x
|
|
{
|
|
"properties": {
|
|
"beat_index": {"type": "integer"}
|
|
}
|
|
}
|
|
|
|
// v2.0.0
|
|
{
|
|
"properties": {
|
|
"beat_index": {"type": "string"}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 3. Removing Enum Values
|
|
|
|
```json
|
|
// v1.x.x
|
|
{
|
|
"properties": {
|
|
"state": {
|
|
"enum": ["idle", "executing", "deprecated_state"]
|
|
}
|
|
}
|
|
}
|
|
|
|
// v2.0.0
|
|
{
|
|
"properties": {
|
|
"state": {
|
|
"enum": ["idle", "executing"]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 4. Tightening Constraints
|
|
|
|
```json
|
|
// v1.x.x
|
|
{
|
|
"properties": {
|
|
"agent_id": {
|
|
"type": "string",
|
|
"maxLength": 256
|
|
}
|
|
}
|
|
}
|
|
|
|
// v2.0.0
|
|
{
|
|
"properties": {
|
|
"agent_id": {
|
|
"type": "string",
|
|
"maxLength": 128
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Patch Version Changes (Non-Breaking)
|
|
|
|
These changes increment the patch version (1.0.0 → 1.0.1):
|
|
|
|
1. **Documentation updates**
|
|
2. **Example additions**
|
|
3. **Description clarifications**
|
|
4. **Comment additions**
|
|
|
|
## Migration Strategies
|
|
|
|
### Minor Version Migration
|
|
|
|
Services automatically benefit from minor version updates:
|
|
|
|
```go
|
|
// This code works with both v1.0.0 and v1.1.0
|
|
func handleBeatFrame(frame BeatFrame) {
|
|
// Core fields always present
|
|
log.Printf("Beat %d in phase %s", frame.BeatIndex, frame.Phase)
|
|
|
|
// New optional fields checked safely
|
|
if frame.Priority != nil {
|
|
log.Printf("Priority: %d", *frame.Priority)
|
|
}
|
|
}
|
|
```
|
|
|
|
### Major Version Migration
|
|
|
|
Requires explicit handling of both versions during transition:
|
|
|
|
```go
|
|
func handleMessage(messageBytes []byte) error {
|
|
var msgType struct {
|
|
Type string `json:"type"`
|
|
}
|
|
|
|
if err := json.Unmarshal(messageBytes, &msgType); err != nil {
|
|
return err
|
|
}
|
|
|
|
switch msgType.Type {
|
|
case "backbeat.beatframe.v1":
|
|
return handleBeatFrameV1(messageBytes)
|
|
case "backbeat.beatframe.v2":
|
|
return handleBeatFrameV2(messageBytes)
|
|
default:
|
|
return fmt.Errorf("unsupported message type: %s", msgType.Type)
|
|
}
|
|
}
|
|
```
|
|
|
|
### Gradual Migration Process
|
|
|
|
1. **Preparation Phase** (Months 1-2)
|
|
- Announce upcoming major version change
|
|
- Publish v2.0.0 schemas alongside v1.x.x
|
|
- Update documentation and examples
|
|
- Provide migration tools and guides
|
|
|
|
2. **Dual Support Phase** (Months 3-4)
|
|
- Services support both v1 and v2 message types
|
|
- New services prefer v2 messages
|
|
- Monitoring tracks v1 vs v2 usage
|
|
|
|
3. **Migration Phase** (Months 5-6)
|
|
- All services updated to send v2 messages
|
|
- Services still accept v1 for backward compatibility
|
|
- Warnings logged for v1 message reception
|
|
|
|
4. **Cleanup Phase** (Month 7+)
|
|
- Drop support for v1 messages
|
|
- Remove v1 handling code
|
|
- Update schemas to mark v1 as deprecated
|
|
|
|
## Implementation Guidelines
|
|
|
|
### Schema Development
|
|
|
|
1. **Start Conservative**: Begin with strict constraints, relax later if needed
|
|
2. **Plan for Growth**: Design extensible structures with optional metadata objects
|
|
3. **Document Thoroughly**: Include clear descriptions and examples
|
|
4. **Test Extensively**: Validate with real-world data before releasing
|
|
|
|
### Version Detection
|
|
|
|
Services should detect schema versions:
|
|
|
|
```go
|
|
type SchemaInfo struct {
|
|
Version string `json:"version"`
|
|
MessageType string `json:"message_type"`
|
|
IsSupported bool `json:"is_supported"`
|
|
}
|
|
|
|
func detectSchemaVersion(messageType string) SchemaInfo {
|
|
switch messageType {
|
|
case "backbeat.beatframe.v1":
|
|
return SchemaInfo{
|
|
Version: "1.x.x",
|
|
MessageType: messageType,
|
|
IsSupported: true,
|
|
}
|
|
case "backbeat.beatframe.v2":
|
|
return SchemaInfo{
|
|
Version: "2.x.x",
|
|
MessageType: messageType,
|
|
IsSupported: true,
|
|
}
|
|
default:
|
|
return SchemaInfo{
|
|
MessageType: messageType,
|
|
IsSupported: false,
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Validation Strategy
|
|
|
|
```go
|
|
func validateWithVersionFallback(messageBytes []byte) error {
|
|
// Try latest version first
|
|
if err := validateV2(messageBytes); err == nil {
|
|
return nil
|
|
}
|
|
|
|
// Fall back to previous version
|
|
if err := validateV1(messageBytes); err == nil {
|
|
log.Warn("Received v1 message, consider upgrading sender")
|
|
return nil
|
|
}
|
|
|
|
return fmt.Errorf("message does not match any supported schema version")
|
|
}
|
|
```
|
|
|
|
## Testing Schema Evolution
|
|
|
|
### Compatibility Tests
|
|
|
|
```go
|
|
func TestSchemaBackwardCompatibility(t *testing.T) {
|
|
// Test that v1.1.0 accepts all valid v1.0.0 messages
|
|
v100Messages := loadTestMessages("v1.0.0")
|
|
v110Schema := loadSchema("beatframe-v1.1.0.schema.json")
|
|
|
|
for _, msg := range v100Messages {
|
|
err := validateAgainstSchema(msg, v110Schema)
|
|
assert.NoError(t, err, "v1.1.0 should accept v1.0.0 messages")
|
|
}
|
|
}
|
|
|
|
func TestSchemaForwardCompatibility(t *testing.T) {
|
|
// Test that v1.0.0 code gracefully handles v1.1.0 messages
|
|
v110Message := loadTestMessage("beatframe-v1.1.0-with-new-fields.json")
|
|
|
|
var beatFrame BeatFrameV1
|
|
err := json.Unmarshal(v110Message, &beatFrame)
|
|
assert.NoError(t, err, "v1.0.0 struct should parse v1.1.0 messages")
|
|
|
|
// Core fields should be populated
|
|
assert.NotEmpty(t, beatFrame.Type)
|
|
assert.NotEmpty(t, beatFrame.ClusterID)
|
|
}
|
|
```
|
|
|
|
### Migration Tests
|
|
|
|
```go
|
|
func TestDualVersionSupport(t *testing.T) {
|
|
handler := NewMessageHandler()
|
|
|
|
v1Message := generateBeatFrameV1()
|
|
v2Message := generateBeatFrameV2()
|
|
|
|
// Both versions should be handled correctly
|
|
err1 := handler.HandleMessage(v1Message)
|
|
err2 := handler.HandleMessage(v2Message)
|
|
|
|
assert.NoError(t, err1)
|
|
assert.NoError(t, err2)
|
|
}
|
|
```
|
|
|
|
## Deprecation Process
|
|
|
|
### Marking Deprecated Features
|
|
|
|
```json
|
|
{
|
|
"properties": {
|
|
"legacy_field": {
|
|
"type": "string",
|
|
"description": "DEPRECATED: Use new_field instead. Will be removed in v2.0.0",
|
|
"deprecated": true
|
|
},
|
|
"new_field": {
|
|
"type": "string",
|
|
"description": "Replacement for legacy_field"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Communication Timeline
|
|
|
|
1. **6 months before**: Announce deprecation in release notes
|
|
2. **3 months before**: Add deprecation warnings to schemas
|
|
3. **1 month before**: Final migration reminder
|
|
4. **Release day**: Remove deprecated features
|
|
|
|
### Tooling Support
|
|
|
|
```bash
|
|
# Check for deprecated schema usage
|
|
backbeat-validate --schemas ./schemas --dir messages/ --check-deprecated
|
|
|
|
# Migration helper
|
|
backbeat-migrate --from v1 --to v2 --dir messages/
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
### For Schema Authors
|
|
|
|
1. **Communicate Early**: Announce changes well in advance
|
|
2. **Provide Tools**: Create migration utilities and documentation
|
|
3. **Monitor Usage**: Track which versions are being used
|
|
4. **Be Conservative**: Prefer minor over major version changes
|
|
|
|
### For Service Developers
|
|
|
|
1. **Stay Updated**: Subscribe to schema change notifications
|
|
2. **Plan for Migration**: Build version handling into your services
|
|
3. **Test Thoroughly**: Validate against multiple schema versions
|
|
4. **Monitor Compatibility**: Alert on unsupported message versions
|
|
|
|
### For Operations Teams
|
|
|
|
1. **Version Tracking**: Monitor which schema versions are active
|
|
2. **Migration Planning**: Coordinate major version migrations
|
|
3. **Rollback Capability**: Be prepared to revert if migrations fail
|
|
4. **Performance Impact**: Monitor schema validation performance
|
|
|
|
## Future Considerations
|
|
|
|
### Planned Enhancements
|
|
|
|
1. **Schema Registry**: Centralized schema version management
|
|
2. **Auto-Migration**: Tools to automatically update message formats
|
|
3. **Version Negotiation**: Services negotiate supported versions
|
|
4. **Schema Analytics**: Usage metrics and compatibility reporting
|
|
|
|
### Long-term Vision
|
|
|
|
- **Continuous Evolution**: Schemas evolve without breaking existing services
|
|
- **Zero-Downtime Updates**: Schema changes deploy without service interruption
|
|
- **Automated Testing**: CI/CD pipelines validate schema compatibility
|
|
- **Self-Healing**: Services automatically adapt to schema changes |