backbeat: add module sources
This commit is contained in:
anthonyrawlins
507
contracts/docs/schema-evolution.md
Normal file
507
contracts/docs/schema-evolution.md
Normal file
@@ -0,0 +1,507 @@
|
||||
# BACKBEAT Schema Evolution and Versioning
|
||||
|
||||
This document defines how BACKBEAT message schemas evolve over time while maintaining compatibility across the CHORUS 2.0.0 ecosystem.
|
||||
|
||||
## Versioning Strategy
|
||||
|
||||
### Semantic Versioning for Schemas
|
||||
|
||||
BACKBEAT schemas follow semantic versioning (SemVer) with CHORUS-specific interpretations:
|
||||
|
||||
- **MAJOR** (`X.0.0`): Breaking changes that require code updates
|
||||
- **MINOR** (`X.Y.0`): Backward-compatible additions (new optional fields, enum values)
|
||||
- **PATCH** (`X.Y.Z`): Documentation updates, constraint clarifications, examples
|
||||
|
||||
### Schema Identification
|
||||
|
||||
Each schema includes version information:
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"$id": "https://chorus.services/schemas/backbeat/beatframe/v1.2.0",
|
||||
"title": "BACKBEAT BeatFrame (INT-A)",
|
||||
"version": "1.2.0"
|
||||
}
|
||||
```
|
||||
|
||||
### Message Type Versioning
|
||||
|
||||
Message types embed version information:
|
||||
|
||||
- `backbeat.beatframe.v1` → Schema version 1.x.x
|
||||
- `backbeat.beatframe.v2` → Schema version 2.x.x
|
||||
|
||||
Only **major** version changes require new message type identifiers.
|
||||
|
||||
## Compatibility Matrix
|
||||
|
||||
### Current Schema Versions
|
||||
|
||||
| Interface | Schema Version | Message Type | Status |
|
||||
|-----------|----------------|--------------|--------|
|
||||
| INT-A (BeatFrame) | 1.0.0 | `backbeat.beatframe.v1` | Active |
|
||||
| INT-B (StatusClaim) | 1.0.0 | `backbeat.statusclaim.v1` | Active |
|
||||
| INT-C (BarReport) | 1.0.0 | `backbeat.barreport.v1` | Active |
|
||||
|
||||
### Version Compatibility Rules
|
||||
|
||||
1. **Minor/Patch Updates**: All v1.x.x schemas are compatible with `backbeat.*.v1` messages
|
||||
2. **Major Updates**: Require new message type (e.g., `backbeat.beatframe.v2`)
|
||||
3. **Transition Period**: Both old and new versions supported during migration
|
||||
4. **Deprecation**: 6-month notice before removing support for old major versions
|
||||
|
||||
## Change Categories
|
||||
|
||||
### Minor Version Changes (Backward Compatible)
|
||||
|
||||
These changes increment the minor version (1.0.0 → 1.1.0):
|
||||
|
||||
#### 1. Adding Optional Fields
|
||||
|
||||
```json
|
||||
// Before (v1.0.0)
|
||||
{
|
||||
"required": ["type", "cluster_id", "beat_index"],
|
||||
"properties": {
|
||||
"type": {...},
|
||||
"cluster_id": {...},
|
||||
"beat_index": {...}
|
||||
}
|
||||
}
|
||||
|
||||
// After (v1.1.0) - adds optional field
|
||||
{
|
||||
"required": ["type", "cluster_id", "beat_index"],
|
||||
"properties": {
|
||||
"type": {...},
|
||||
"cluster_id": {...},
|
||||
"beat_index": {...},
|
||||
"priority": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"maximum": 10,
|
||||
"description": "Optional processing priority (1=low, 10=high)"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 2. Adding Enum Values
|
||||
|
||||
```json
|
||||
// Before (v1.0.0)
|
||||
{
|
||||
"properties": {
|
||||
"phase": {
|
||||
"enum": ["plan", "execute", "review"]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// After (v1.1.0) - adds new phase
|
||||
{
|
||||
"properties": {
|
||||
"phase": {
|
||||
"enum": ["plan", "execute", "review", "cleanup"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. Relaxing Constraints
|
||||
|
||||
```json
|
||||
// Before (v1.0.0)
|
||||
{
|
||||
"properties": {
|
||||
"notes": {
|
||||
"type": "string",
|
||||
"maxLength": 256
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// After (v1.1.0) - allows longer notes
|
||||
{
|
||||
"properties": {
|
||||
"notes": {
|
||||
"type": "string",
|
||||
"maxLength": 512
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 4. Adding Properties to Objects
|
||||
|
||||
```json
|
||||
// Before (v1.0.0)
|
||||
{
|
||||
"properties": {
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"version": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// After (v1.1.0) - adds new metadata field
|
||||
{
|
||||
"properties": {
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"version": {"type": "string"},
|
||||
"source": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Major Version Changes (Breaking)
|
||||
|
||||
These changes increment the major version (1.x.x → 2.0.0):
|
||||
|
||||
#### 1. Removing Required Fields
|
||||
|
||||
```json
|
||||
// v1.x.x
|
||||
{
|
||||
"required": ["type", "cluster_id", "beat_index", "deprecated_field"]
|
||||
}
|
||||
|
||||
// v2.0.0
|
||||
{
|
||||
"required": ["type", "cluster_id", "beat_index"]
|
||||
}
|
||||
```
|
||||
|
||||
#### 2. Changing Field Types
|
||||
|
||||
```json
|
||||
// v1.x.x
|
||||
{
|
||||
"properties": {
|
||||
"beat_index": {"type": "integer"}
|
||||
}
|
||||
}
|
||||
|
||||
// v2.0.0
|
||||
{
|
||||
"properties": {
|
||||
"beat_index": {"type": "string"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. Removing Enum Values
|
||||
|
||||
```json
|
||||
// v1.x.x
|
||||
{
|
||||
"properties": {
|
||||
"state": {
|
||||
"enum": ["idle", "executing", "deprecated_state"]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// v2.0.0
|
||||
{
|
||||
"properties": {
|
||||
"state": {
|
||||
"enum": ["idle", "executing"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 4. Tightening Constraints
|
||||
|
||||
```json
|
||||
// v1.x.x
|
||||
{
|
||||
"properties": {
|
||||
"agent_id": {
|
||||
"type": "string",
|
||||
"maxLength": 256
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// v2.0.0
|
||||
{
|
||||
"properties": {
|
||||
"agent_id": {
|
||||
"type": "string",
|
||||
"maxLength": 128
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Patch Version Changes (Non-Breaking)
|
||||
|
||||
These changes increment the patch version (1.0.0 → 1.0.1):
|
||||
|
||||
1. **Documentation updates**
|
||||
2. **Example additions**
|
||||
3. **Description clarifications**
|
||||
4. **Comment additions**
|
||||
|
||||
## Migration Strategies
|
||||
|
||||
### Minor Version Migration
|
||||
|
||||
Services automatically benefit from minor version updates:
|
||||
|
||||
```go
|
||||
// This code works with both v1.0.0 and v1.1.0
|
||||
func handleBeatFrame(frame BeatFrame) {
|
||||
// Core fields always present
|
||||
log.Printf("Beat %d in phase %s", frame.BeatIndex, frame.Phase)
|
||||
|
||||
// New optional fields checked safely
|
||||
if frame.Priority != nil {
|
||||
log.Printf("Priority: %d", *frame.Priority)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Major Version Migration
|
||||
|
||||
Requires explicit handling of both versions during transition:
|
||||
|
||||
```go
|
||||
func handleMessage(messageBytes []byte) error {
|
||||
var msgType struct {
|
||||
Type string `json:"type"`
|
||||
}
|
||||
|
||||
if err := json.Unmarshal(messageBytes, &msgType); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
switch msgType.Type {
|
||||
case "backbeat.beatframe.v1":
|
||||
return handleBeatFrameV1(messageBytes)
|
||||
case "backbeat.beatframe.v2":
|
||||
return handleBeatFrameV2(messageBytes)
|
||||
default:
|
||||
return fmt.Errorf("unsupported message type: %s", msgType.Type)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Gradual Migration Process
|
||||
|
||||
1. **Preparation Phase** (Months 1-2)
|
||||
- Announce upcoming major version change
|
||||
- Publish v2.0.0 schemas alongside v1.x.x
|
||||
- Update documentation and examples
|
||||
- Provide migration tools and guides
|
||||
|
||||
2. **Dual Support Phase** (Months 3-4)
|
||||
- Services support both v1 and v2 message types
|
||||
- New services prefer v2 messages
|
||||
- Monitoring tracks v1 vs v2 usage
|
||||
|
||||
3. **Migration Phase** (Months 5-6)
|
||||
- All services updated to send v2 messages
|
||||
- Services still accept v1 for backward compatibility
|
||||
- Warnings logged for v1 message reception
|
||||
|
||||
4. **Cleanup Phase** (Month 7+)
|
||||
- Drop support for v1 messages
|
||||
- Remove v1 handling code
|
||||
- Update schemas to mark v1 as deprecated
|
||||
|
||||
## Implementation Guidelines
|
||||
|
||||
### Schema Development
|
||||
|
||||
1. **Start Conservative**: Begin with strict constraints, relax later if needed
|
||||
2. **Plan for Growth**: Design extensible structures with optional metadata objects
|
||||
3. **Document Thoroughly**: Include clear descriptions and examples
|
||||
4. **Test Extensively**: Validate with real-world data before releasing
|
||||
|
||||
### Version Detection
|
||||
|
||||
Services should detect schema versions:
|
||||
|
||||
```go
|
||||
type SchemaInfo struct {
|
||||
Version string `json:"version"`
|
||||
MessageType string `json:"message_type"`
|
||||
IsSupported bool `json:"is_supported"`
|
||||
}
|
||||
|
||||
func detectSchemaVersion(messageType string) SchemaInfo {
|
||||
switch messageType {
|
||||
case "backbeat.beatframe.v1":
|
||||
return SchemaInfo{
|
||||
Version: "1.x.x",
|
||||
MessageType: messageType,
|
||||
IsSupported: true,
|
||||
}
|
||||
case "backbeat.beatframe.v2":
|
||||
return SchemaInfo{
|
||||
Version: "2.x.x",
|
||||
MessageType: messageType,
|
||||
IsSupported: true,
|
||||
}
|
||||
default:
|
||||
return SchemaInfo{
|
||||
MessageType: messageType,
|
||||
IsSupported: false,
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Validation Strategy
|
||||
|
||||
```go
|
||||
func validateWithVersionFallback(messageBytes []byte) error {
|
||||
// Try latest version first
|
||||
if err := validateV2(messageBytes); err == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Fall back to previous version
|
||||
if err := validateV1(messageBytes); err == nil {
|
||||
log.Warn("Received v1 message, consider upgrading sender")
|
||||
return nil
|
||||
}
|
||||
|
||||
return fmt.Errorf("message does not match any supported schema version")
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Schema Evolution
|
||||
|
||||
### Compatibility Tests
|
||||
|
||||
```go
|
||||
func TestSchemaBackwardCompatibility(t *testing.T) {
|
||||
// Test that v1.1.0 accepts all valid v1.0.0 messages
|
||||
v100Messages := loadTestMessages("v1.0.0")
|
||||
v110Schema := loadSchema("beatframe-v1.1.0.schema.json")
|
||||
|
||||
for _, msg := range v100Messages {
|
||||
err := validateAgainstSchema(msg, v110Schema)
|
||||
assert.NoError(t, err, "v1.1.0 should accept v1.0.0 messages")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSchemaForwardCompatibility(t *testing.T) {
|
||||
// Test that v1.0.0 code gracefully handles v1.1.0 messages
|
||||
v110Message := loadTestMessage("beatframe-v1.1.0-with-new-fields.json")
|
||||
|
||||
var beatFrame BeatFrameV1
|
||||
err := json.Unmarshal(v110Message, &beatFrame)
|
||||
assert.NoError(t, err, "v1.0.0 struct should parse v1.1.0 messages")
|
||||
|
||||
// Core fields should be populated
|
||||
assert.NotEmpty(t, beatFrame.Type)
|
||||
assert.NotEmpty(t, beatFrame.ClusterID)
|
||||
}
|
||||
```
|
||||
|
||||
### Migration Tests
|
||||
|
||||
```go
|
||||
func TestDualVersionSupport(t *testing.T) {
|
||||
handler := NewMessageHandler()
|
||||
|
||||
v1Message := generateBeatFrameV1()
|
||||
v2Message := generateBeatFrameV2()
|
||||
|
||||
// Both versions should be handled correctly
|
||||
err1 := handler.HandleMessage(v1Message)
|
||||
err2 := handler.HandleMessage(v2Message)
|
||||
|
||||
assert.NoError(t, err1)
|
||||
assert.NoError(t, err2)
|
||||
}
|
||||
```
|
||||
|
||||
## Deprecation Process
|
||||
|
||||
### Marking Deprecated Features
|
||||
|
||||
```json
|
||||
{
|
||||
"properties": {
|
||||
"legacy_field": {
|
||||
"type": "string",
|
||||
"description": "DEPRECATED: Use new_field instead. Will be removed in v2.0.0",
|
||||
"deprecated": true
|
||||
},
|
||||
"new_field": {
|
||||
"type": "string",
|
||||
"description": "Replacement for legacy_field"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Communication Timeline
|
||||
|
||||
1. **6 months before**: Announce deprecation in release notes
|
||||
2. **3 months before**: Add deprecation warnings to schemas
|
||||
3. **1 month before**: Final migration reminder
|
||||
4. **Release day**: Remove deprecated features
|
||||
|
||||
### Tooling Support
|
||||
|
||||
```bash
|
||||
# Check for deprecated schema usage
|
||||
backbeat-validate --schemas ./schemas --dir messages/ --check-deprecated
|
||||
|
||||
# Migration helper
|
||||
backbeat-migrate --from v1 --to v2 --dir messages/
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### For Schema Authors
|
||||
|
||||
1. **Communicate Early**: Announce changes well in advance
|
||||
2. **Provide Tools**: Create migration utilities and documentation
|
||||
3. **Monitor Usage**: Track which versions are being used
|
||||
4. **Be Conservative**: Prefer minor over major version changes
|
||||
|
||||
### For Service Developers
|
||||
|
||||
1. **Stay Updated**: Subscribe to schema change notifications
|
||||
2. **Plan for Migration**: Build version handling into your services
|
||||
3. **Test Thoroughly**: Validate against multiple schema versions
|
||||
4. **Monitor Compatibility**: Alert on unsupported message versions
|
||||
|
||||
### For Operations Teams
|
||||
|
||||
1. **Version Tracking**: Monitor which schema versions are active
|
||||
2. **Migration Planning**: Coordinate major version migrations
|
||||
3. **Rollback Capability**: Be prepared to revert if migrations fail
|
||||
4. **Performance Impact**: Monitor schema validation performance
|
||||
|
||||
## Future Considerations
|
||||
|
||||
### Planned Enhancements
|
||||
|
||||
1. **Schema Registry**: Centralized schema version management
|
||||
2. **Auto-Migration**: Tools to automatically update message formats
|
||||
3. **Version Negotiation**: Services negotiate supported versions
|
||||
4. **Schema Analytics**: Usage metrics and compatibility reporting
|
||||
|
||||
### Long-term Vision
|
||||
|
||||
- **Continuous Evolution**: Schemas evolve without breaking existing services
|
||||
- **Zero-Downtime Updates**: Schema changes deploy without service interruption
|
||||
- **Automated Testing**: CI/CD pipelines validate schema compatibility
|
||||
- **Self-Healing**: Services automatically adapt to schema changes
|
||||
Reference in New Issue
Block a user