chorus-services/planning and reports/DATABASE_INTEGRATION.md

CHORUS Website Database Integration - Brand Updated

This document describes the updated CHORUS website database integration aligned with current brand guidelines and technical specifications, including lead capture system optimization for ultra-minimalist design and accessibility compliance.

Integration Overview

The website module now uses the centralized PostgreSQL database service instead of maintaining its own separate database. This provides:

• Unified Infrastructure: Single PostgreSQL instance for all CHORUS services
• Simplified Management: One database to backup, monitor, and maintain
• Better Resource Utilization: Shared database resources across services
• Consistent Configuration: Same database credentials and network across all services

Database Architecture

Main Database Service

• Host: postgres (Docker service name)
• Port: 5432 (internal), 5433 (external)
• User: chorus
• Password: choruspass

Website Database (Brand-Optimized)

• Database Name: chorus_website
• Connection String: postgresql://chorus:choruspass@postgres:5432/chorus_website
• Performance Optimization: Optimized for ultra-minimalist UI with fast query response times
• Accessibility Integration: Stores user accessibility theme preferences and interaction data

Updated Database Structure (Brand-Aligned)

chorus_postgres (PostgreSQL 15 container)
├── chorus_whoosh      (WHOOSH orchestration data)
├── chorus_slurp       (SLURP context curation data)
├── chorus_rl_tuner    (RL feedback system data)
├── chorus_monitoring  (Monitoring and metrics data)
└── chorus_website     (Website lead capture & brand data)
    ├── leads                    (Main lead capture table)
    ├── consent_audit           (GDPR consent tracking)
    ├── lead_interactions       (CRM interaction history)
    ├── accessibility_preferences (User accessibility themes)
    ├── website_analytics       (Minimalist analytics data)
    ├── brand_engagement        (Three.js logo interaction data)
    ├── rate_limits            (Spam protection)
    ├── data_retention_schedule (GDPR data cleanup)
    └── schema_migrations      (Migration tracking)

Migration System

Migration Files

• Location: /modules/website/database/migrations/
• Naming: 001_initial_lead_capture.sql, 002_next_feature.sql, etc.
• Tracking: schema_migrations table tracks applied migrations

Running Migrations

# Automated migration runner
cd /modules/website/database/
./run-migrations.sh

# Manual migration (if needed)
psql postgresql://chorus:choruspass@postgres:5432/chorus_website -f migrations/001_initial_lead_capture.sql

Integration Setup

1. Database Initialization

The main init-db.sql now includes:

CREATE DATABASE chorus_website;
GRANT ALL PRIVILEGES ON DATABASE chorus_website TO chorus;

\c chorus_website;
CREATE TABLE schema_migrations (
    version VARCHAR(255) PRIMARY KEY,
    applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

2. Website Configuration Updates

• API Route: Updated to use chorus_website database
• Docker Compose: Connected to external chorus_network
• Environment: Database URL points to main postgres service

3. Network Integration

# Website connects to main CHORUS network
networks:
  - chorus-dev                    # Local development
  - chorus-services_chorus_network # Main CHORUS network

Development Workflow

Starting the Integrated System

1. Start Main CHORUS Infrastructure:

   cd /chorus.services/
   docker-compose up -d postgres redis
   

2. Run Website Integration Setup:

   cd /chorus.services/modules/website/
   ./integrate-with-chorus.sh
   

3. Start Website Development:

   # Option 1: Standalone development
   docker-compose up website-dev
   
   # Option 2: With database profile
   docker-compose --profile database up website-dev
   

Production Deployment

1. Full CHORUS Stack with Website:

   cd /chorus.services/
   docker-compose --profile website up -d
   

2. Website Only (assumes main stack is running):

   docker-compose up -d chorus-website
   

Database Schema Features (Brand-Enhanced)

Lead Capture Tables

leads (Enhanced for Brand Compliance)

CREATE TABLE leads (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    first_name VARCHAR(100) NOT NULL,
    last_name VARCHAR(100) NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL,
    company_name VARCHAR(200),
    inquiry_details TEXT,
    lead_source VARCHAR(50) DEFAULT 'contact_form',
    accessibility_theme VARCHAR(20) DEFAULT 'default', -- Brand accessibility preference
    preferred_communication VARCHAR(20) DEFAULT 'email',
    gdpr_consent BOOLEAN NOT NULL DEFAULT FALSE,
    marketing_consent BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW(),
    
    -- Brand engagement tracking
    logo_interactions INTEGER DEFAULT 0,
    time_on_site INTEGER, -- milliseconds for performance analysis
    pages_viewed JSONB DEFAULT '[]'::jsonb,
    
    -- Performance and analytics
    load_time_lcp INTEGER, -- Largest Contentful Paint
    load_time_fid INTEGER, -- First Input Delay
    device_type VARCHAR(20), -- mobile, tablet, desktop
    browser_info JSONB
);

-- Indexes for performance
CREATE INDEX idx_leads_created_at ON leads(created_at);
CREATE INDEX idx_leads_accessibility_theme ON leads(accessibility_theme);
CREATE INDEX idx_leads_lead_source ON leads(lead_source);

accessibility_preferences (New Brand Table)

CREATE TABLE accessibility_preferences (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    session_id VARCHAR(100),
    accessibility_theme VARCHAR(20) NOT NULL, -- default, protanopia, deuteranopia, tritanopia, achromatopsia
    dark_mode_preference BOOLEAN DEFAULT TRUE, -- Dark mode default per brand
    reduced_motion BOOLEAN DEFAULT FALSE,
    font_size_preference VARCHAR(10) DEFAULT 'medium',
    high_contrast BOOLEAN DEFAULT FALSE,
    
    -- Brand interaction data
    logo_interaction_count INTEGER DEFAULT 0,
    theme_switch_count INTEGER DEFAULT 0,
    session_duration INTEGER, -- milliseconds
    
    created_at TIMESTAMPTZ DEFAULT NOW(),
    last_updated TIMESTAMPTZ DEFAULT NOW()
);

-- Performance indexes
CREATE INDEX idx_accessibility_session ON accessibility_preferences(session_id);
CREATE INDEX idx_accessibility_theme ON accessibility_preferences(accessibility_theme);

consent_audit (Enhanced)

• Brand Compliance: Tracks accessibility theme consent changes
• Three.js Interaction: Logs consent for logo interaction data collection
• Performance Analytics: Consent for collecting Core Web Vitals data

brand_engagement (New Table)

CREATE TABLE brand_engagement (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    session_id VARCHAR(100),
    lead_id UUID REFERENCES leads(id),
    
    -- Three.js logo interactions
    logo_interactions JSONB DEFAULT '[]'::jsonb,
    logo_accessibility_theme_changes INTEGER DEFAULT 0,
    
    -- Ultra-minimalist design engagement
    scroll_depth_percentage INTEGER,
    time_spent_per_section JSONB DEFAULT '{}'::jsonb,
    cta_clicks JSONB DEFAULT '[]'::jsonb,
    
    -- Performance metrics
    core_web_vitals JSONB DEFAULT '{}'::jsonb,
    page_load_performance JSONB DEFAULT '{}'::jsonb,
    
    -- Accessibility usage
    accessibility_features_used JSONB DEFAULT '[]'::jsonb,
    keyboard_navigation_used BOOLEAN DEFAULT FALSE,
    
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_brand_engagement_session ON brand_engagement(session_id);
CREATE INDEX idx_brand_engagement_lead ON brand_engagement(lead_id);

lead_interactions (Enhanced)

• Brand-Aware CRM: Tracks accessibility preferences in interactions
• Performance Context: Records page performance during interactions
• Design Feedback: Captures ultra-minimalist design effectiveness

rate_limits

• Spam Protection: IP and email-based rate limiting
• Configurable Limits: Different limits per action type
• Automatic Cleanup: Expired limits are cleared

Security Features

• Input Validation: Zod schema validation
• SQL Injection Protection: Parameterized queries
• Rate Limiting: Multiple rate limiters (IP, email)
• GDPR Compliance: Consent tracking, data retention
• Phone Validation: International phone number formatting

API Endpoints (Brand-Enhanced)

POST /api/leads (Updated)

Lead capture endpoint with brand compliance and accessibility support.

Enhanced Request Example:

curl -X POST http://localhost:3000/api/leads \
  -H 'Content-Type: application/json' \
  -d '{
    "firstName": "John",
    "lastName": "Doe",
    "email": "john.doe@example.com",
    "leadSource": "contact_form",
    "gdprConsent": true,
    "companyName": "Acme Corp",
    "inquiryDetails": "Interested in technical deep dive",
    
    // Brand-specific fields
    "accessibilityTheme": "protanopia",
    "deviceType": "desktop",
    "coreWebVitals": {
      "lcp": 2300,
      "fid": 85,
      "cls": 0.08
    },
    "logoInteractions": 3,
    "timeOnSite": 145000,
    "pagesViewed": ["/", "/ecosystem", "/contact"]
  }'

POST /api/accessibility-preferences

// New endpoint for storing user accessibility preferences
interface AccessibilityPreferencesRequest {
  sessionId: string;
  accessibilityTheme: 'default' | 'protanopia' | 'deuteranopia' | 'tritanopia' | 'achromatopsia';
  darkModePreference: boolean;
  reducedMotion: boolean;
  logoInteractionCount: number;
  sessionDuration: number;
}

// Usage
curl -X POST http://localhost:3000/api/accessibility-preferences \
  -H 'Content-Type: application/json' \
  -d '{
    "sessionId": "sess_abc123",
    "accessibilityTheme": "deuteranopia",
    "darkModePreference": true,
    "reducedMotion": false,
    "logoInteractionCount": 5,
    "sessionDuration": 180000
  }'

POST /api/brand-engagement

// Track Three.js logo and brand element interactions
interface BrandEngagementRequest {
  sessionId: string;
  logoInteractions: Array<{timestamp: number, theme: string, duration: number}>;
  scrollDepthPercentage: number;
  timeSpentPerSection: Record<string, number>;
  coreWebVitals: {lcp: number, fid: number, cls: number};
  accessibilityFeaturesUsed: string[];
}

GET /api/leads (Enhanced)

Protected endpoint with brand analytics (requires API key).

Monitoring and Maintenance (Brand-Enhanced)

Database Monitoring

-- Performance monitoring queries for brand compliance

-- Track accessibility theme usage
SELECT 
    accessibility_theme,
    COUNT(*) as usage_count,
    AVG(session_duration) as avg_session_duration
FROM accessibility_preferences 
WHERE created_at >= NOW() - INTERVAL '30 days'
GROUP BY accessibility_theme;

-- Monitor Three.js logo performance impact
SELECT 
    AVG(core_web_vitals->>'lcp')::numeric as avg_lcp,
    AVG(core_web_vitals->>'fid')::numeric as avg_fid,
    AVG(core_web_vitals->>'cls')::numeric as avg_cls
FROM brand_engagement 
WHERE created_at >= NOW() - INTERVAL '7 days';

-- Track ultra-minimalist design effectiveness
SELECT 
    AVG(scroll_depth_percentage) as avg_scroll_depth,
    AVG(JSON_ARRAY_LENGTH(cta_clicks)) as avg_cta_interactions
FROM brand_engagement
WHERE created_at >= NOW() - INTERVAL '30 days';

GDPR Compliance (Brand-Enhanced)

-- Enhanced GDPR compliance with brand data

-- Data retention with accessibility preferences
CREATE OR REPLACE FUNCTION cleanup_expired_brand_data()
RETURNS void AS $$
BEGIN
    -- Clean up old accessibility preferences (90 days)
    DELETE FROM accessibility_preferences 
    WHERE created_at < NOW() - INTERVAL '90 days'
    AND session_id NOT IN (SELECT DISTINCT session_id FROM leads);
    
    -- Clean up brand engagement data (2 years)
    DELETE FROM brand_engagement 
    WHERE created_at < NOW() - INTERVAL '2 years'
    AND lead_id IS NULL;
    
    -- Standard GDPR cleanup for leads (7 years)
    DELETE FROM leads 
    WHERE created_at < NOW() - INTERVAL '7 years'
    AND gdpr_consent = false;
END;
$$ LANGUAGE plpgsql;

-- Schedule cleanup
SELECT cron.schedule('brand-data-cleanup', '0 2 * * 0', 'SELECT cleanup_expired_brand_data();');

Backup Strategy

The website data is automatically included in the main CHORUS database backup strategy:

• Full Backups: Daily backups of entire PostgreSQL instance
• Point-in-time Recovery: Transaction log shipping
• Cross-site Replication: For disaster recovery

Troubleshooting

Common Issues

1. Connection Refused

   # Ensure main CHORUS postgres is running
   docker-compose up -d postgres
   
   # Check network connectivity
   docker network ls | grep chorus
   

2. Database Not Found

   # Recreate postgres container to run init-db.sql
   docker-compose down postgres
   docker-compose up -d postgres
   

3. Migration Failures

   # Check migration status
   psql postgresql://chorus:choruspass@localhost:5433/chorus_website \
     -c "SELECT * FROM schema_migrations;"
   
   # Run migrations manually
   cd database/
   ./run-migrations.sh
   

Brand-Aware Health Checks

# Test database connectivity with brand tables
psql postgresql://chorus:choruspass@localhost:5433/chorus_website -c "
  SELECT 
    'leads' as table_name, COUNT(*) as row_count 
  FROM leads 
  UNION ALL
  SELECT 
    'accessibility_preferences', COUNT(*) 
  FROM accessibility_preferences
  UNION ALL
  SELECT 
    'brand_engagement', COUNT(*) 
  FROM brand_engagement;"

# Test brand-enhanced API endpoints
curl -f http://localhost:3000/api/leads/health
curl -f http://localhost:3000/api/accessibility-preferences/health
curl -f http://localhost:3000/api/brand-engagement/health

# Check Three.js logo performance impact
curl -f http://localhost:3000/api/brand-analytics/performance

# Verify accessibility theme data collection
curl -f http://localhost:3000/api/brand-analytics/accessibility-usage

Security Considerations

• Network Isolation: Website container only has access to necessary services
• Credential Management: Database credentials managed via environment variables
• Input Validation: Multi-layer validation (client, API, database)
• Rate Limiting: Protection against abuse and spam
• GDPR Compliance: Full consent management and data retention policies

Future Enhancements (Brand-Focused)

Performance Optimization

1. Three.js Logo Optimization: Database caching for logo material preferences
2. Real-time Analytics: WebSocket integration for live brand engagement metrics
3. Accessibility Analytics: ML-based analysis of accessibility theme effectiveness

Brand Intelligence

-- Planned brand intelligence features

-- A/B testing for ultra-minimalist design elements
CREATE TABLE brand_ab_tests (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    test_name VARCHAR(100) NOT NULL,
    variant_a_design JSONB,
    variant_b_design JSONB,
    conversion_metrics JSONB,
    accessibility_impact JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Logo interaction heatmaps
CREATE TABLE logo_interaction_heatmaps (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    session_id VARCHAR(100),
    interaction_coordinates JSONB, -- x,y coordinates of interactions
    accessibility_theme VARCHAR(20),
    interaction_duration INTEGER,
    timestamp TIMESTAMPTZ DEFAULT NOW()
);

-- Performance benchmarking
CREATE TABLE performance_benchmarks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    page_path VARCHAR(200),
    device_type VARCHAR(20),
    accessibility_theme VARCHAR(20),
    core_web_vitals JSONB,
    lighthouse_scores JSONB,
    three_js_render_time INTEGER,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

Advanced Features

1. Connection Pooling: pgBouncer with brand-optimized connection management
2. Read Replicas: Dedicated replicas for accessibility and brand analytics
3. Real-time Monitoring: Prometheus metrics for Three.js logo performance
4. Brand Audit Logging: Complete audit trail for all brand-related data changes
5. Encryption: Field-level encryption for accessibility preferences and engagement data