tony/CHORUS
1
0
Fork 0
Code Issues 10 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7d1a64d8058003e5116b86babde6807ebbb3a305
CHORUS/docs/distos/councils/01-process-scheduling.md

30 KiB
Raw Blame History

Council Design Brief: Process Scheduling

Council ID: council-sched Mission: Design the heterogeneous process scheduling subsystem for DistOS, covering GPU kernel dispatch, workload placement across Hopper/Grace/Blackwell accelerators, fair multi-tenant queuing, gang scheduling for distributed training, and energy-aware execution across a 1024-node cluster. UCXL Base Address: ucxl://council-sched:*@DistOS:scheduling/* Agent Count: 80 Status: Constitution Phase — awaiting WHOOSH formation trigger Created: 2026-02-24

1. Scope and Responsibilities

council-sched owns the complete specification of the DistOS process and kernel scheduling subsystem. Scope boundaries are defined as follows.

In scope:

  • GPU kernel scheduling: dispatch queue management, kernel concurrency, SM partitioning on Hopper (MIG) and Blackwell, and MPS (Multi-Process Service) lifetime management
  • CPU-GPU co-scheduling on Grace Superchip (NVLink-C2C coherent interconnect), including unified virtual address space scheduling implications
  • Workload placement policy across heterogeneous accelerator types (H100, GH200, B200), including topology-aware affinity scoring
  • Fair multi-tenant queuing: priority classes, weighted fair queuing, and quota enforcement
  • Gang scheduling for distributed training workloads: all-or-nothing allocation, partial-allotment strategies, and backfill
  • Preemption strategies: checkpoint-based preemption, time-sliced preemption, and priority inversion avoidance
  • NUMA-aware placement across CPU sockets and GPU memory domains
  • GPU memory oversubscription scheduling: eviction policy, swap-to-host, and coordinated demand management with council-mem
  • Energy-aware scheduling: frequency/voltage scaling directives, power capping, and thermal headroom management
  • Formal specification of the scheduling API surface exposed to user workloads and to other DistOS subsystems

Out of scope (delegated):

  • Physical memory allocation and coherence protocol (delegated to council-mem)
  • Network topology discovery and network-aware placement data (delegated to council-net; consumed as inputs)
  • Metering, cost attribution, and SLO enforcement (delegated to council-telemetry; consumed as outputs)
  • Security isolation between tenants at the hardware level (delegated to council-sec; policies consumed as constraints)

2. Research Domains

2.1 GPU Kernel Scheduling and SM Partitioning

Survey the CUDA Concurrent Kernel Execution model, the NVIDIA Multi-Instance GPU (MIG) architecture on Hopper (H100), and the NVIDIA Multi-Process Service (MPS). Understand how MIG partitions a GPU into isolated GPC slices with dedicated HBM3 memory, and how MPS enables concurrent kernel execution within a single GPU context without full MIG isolation overhead.

Key materials:

  • NVIDIA H100 Architecture Technical Overview (2022) — SM partitioning, GPC layout, NVLink 4.0 bandwidth
  • NVIDIA MIG User Guide (CUDA 12.x) — partition profiles (1g.10gb through 7g.80gb), instance isolation, compute and memory capacity tables
  • NVIDIA MPS documentation — shared context model, client limit (48 for Hopper), error containment limitations
  • AMD ROCm Hardware Abstraction Layer (HAL) source — amdkfd KFD driver, compute queue management, HWS (Hardware Scheduler) in GFX12
  • Jain et al., "Fractional GPUs: Software-Based Compute and Memory Bandwidth Reservation for GPUs" (RTAS 2019) — software-defined partitioning as a baseline comparison
  • Xiao et al., "AntMan: Dynamic Scaling on GPU Clusters for Deep Learning" (OSDI 2020) — GPU memory and compute sharing for co-located workloads

2.2 Heterogeneous Workload Placement

Develop a placement model that accounts for the distinct performance characteristics of H100 (PCIe/SXM5), GH200 Grace Superchip (NVLink-C2C), and B200 Blackwell (NVLink 5.0 SXM). Each accelerator type has different compute density, memory bandwidth, and interconnect topology.

Key materials:

  • Jouppi et al., "In-Datacenter Performance Analysis of a Tensor Processing Unit" (ISCA 2017) — heterogeneous accelerator placement rationale
  • Google Borg paper: Verma et al., "Large-scale cluster management at Google with Borg" (EuroSys 2015) — machine heterogeneity handling, alloc sets, resource estimation
  • Microsoft Singularity: Qiao et al., "Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning" (OSDI 2021) — goodput-aware placement for training workloads
  • Weng et al., "MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters" (NSDI 2022) — real-world heterogeneous GPU cluster scheduling from Alibaba

2.3 Fair Multi-Tenant Queuing

Design a multi-level queueing architecture with Dominant Resource Fairness (DRF) semantics extended for GPU resources (SM fraction, HBM bandwidth, NVLink bandwidth as co-dominant resources).

Key materials:

  • Ghodsi et al., "Dominant Resource Fairness: Fair Allocation of Multiple Resource Types" (NSDI 2011) — foundational DRF theory
  • Apache YARN CapacityScheduler and FairScheduler documentation — hierarchical queues, preemption policy, label-based node targeting
  • Apache Mesos DRF implementation — DominantShareAllocator, offer model, role weights
  • Kubernetes device plugin framework (k8s.io/device-plugins) — GPU resource advertisement, extended resource scheduling
  • Tiresias: Gu et al., "Tiresias: A GPU Cluster Manager for Distributed Deep Learning" (NSDI 2019) — LAS (Least Attained Service) scheduling for DL jobs, 2DAS (2-dimensional attained service)
  • Gandiva: Xiao et al., "Gandiva: Introspective Cluster Scheduling for Deep Learning" (OSDI 2018) — time-sliced GPU sharing, job packing, migration

2.4 Gang Scheduling for Distributed Training

Gang scheduling ensures all processes in a distributed training job (e.g., a 512-GPU allreduce ring) are co-scheduled simultaneously. This is critical for preventing head-of-line blocking and deadlock in collective communication.

Key materials:

  • Feitelson and Rudolph, "Gang Scheduling Performance Benefits for Fine-Grain Synchronization" (J. Parallel Distrib. Comput., 1992) — foundational gang scheduling analysis
  • Rajachandrasekar et al., "A Closer Look at All-Reduce for Deep Learning" (Workshop at SC'19) — collective communication scheduling sensitivity
  • Hwang et al., "AFS: Annotation-Free Automatic Sharding for Large Language Models" (ICLR 2023) — pipeline and tensor parallel placement co-scheduling
  • Slurm gang scheduling documentation — Oversubscribe=FORCE, GraceTime, slurmctld gang plugin
  • Jeon et al., "Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads" (USENIX ATC 2019) — Microsoft Philly cluster trace, gang failure modes

2.5 Preemption Strategies

Survey checkpoint-based, reactive, and time-sliced preemption. Understand the cost of GPU checkpoint (saving SM register file, shared memory, and in-flight DMA) and design preemption policies that bound maximum latency impact.

Key materials:

  • Park et al., "Preemptive, Low Latency Datacenter Scheduling via Lightweight Virtualization" (USENIX ATC 2020) — container-level preemption
  • Lin et al., "SHEPHERD: Serving DNNs in the Wild" (NSDI 2023) — latency SLO-aware preemption for inference workloads
  • NVIDIA CUDA checkpoint/restore (CRIU for GPU) — experimental support status as of CUDA 12.x
  • Wang et al., "Achieving Microsecond-Scale Tail Latency Efficiently with Approximate Optimal Scheduling" (SOSP 2017) — preemption-aware scheduling theory

2.6 NUMA-Aware and Topology-Aware Placement

Model the NUMA topology of a Grace Superchip node: 72-core Arm Neoverse V2 CPU, 480 GB LPDDR5X at ~512 GB/s, connected to H100 GPU via NVLink-C2C at 900 GB/s. Compare this with standard SXM5 nodes where CPU-GPU bandwidth is limited to PCIe Gen5 (~128 GB/s bidirectional).

Key materials:

  • Linux NUMA scheduling documentation — libnuma, numactl, CFS NUMA balancing (task_numa_migrate)
  • NVIDIA GH200 Grace Hopper Superchip Architecture Whitepaper (2023)
  • Lepers et al., "Thread and Memory Placement on NUMA Systems: Asymmetry Matters" (USENIX ATC 2015)
  • Blagodurov et al., "A Case for NUMA-Aware Contention Management on Multicore Systems" (USENIX ATC 2011)

2.7 GPU Memory Oversubscription

When aggregate GPU memory demand exceeds physical HBM3 capacity, the scheduler must coordinate with council-mem's eviction and tiering policies to decide which kernels to throttle, checkpoint, or migrate.

Key materials:

  • Rhu et al., "vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design" (MICRO 2016) — layer-by-layer activation offload
  • Huang et al., "Efficient Large Scale Language Modeling with Mixtures of Experts" (EMNLP 2021) — memory-efficient model parallelism
  • NVIDIA Unified Memory documentation (CUDA 12.x) — page migration engine, oversubscription, prefetch advisory API
  • Kumar et al., "SwapAdvisor: Push Deep Learning Beyond the GPU Memory Limit via Smart Swapping" (ASPLOS 2020)

2.8 Energy-Aware Scheduling

Design scheduling policies that honour per-node power caps enforced by Baseboard Management Controllers (BMCs) and exploit DVFS (Dynamic Voltage and Frequency Scaling) headroom reported by council-telemetry.

Key materials:

  • Patel et al., "Clite: Efficient and QoS Aware Co-location of Multiple Latency-Critical Jobs for Warehouse Scale Computers" (ISCA 2020)
  • Lim et al., "Adaptive Power Management for Heterogeneous Multiprocessor SoCs" (ICCAD 2009)
  • NVIDIA NVML power management API — nvmlDeviceSetPowerManagementLimit, nvmlDeviceGetCurrentClocksThrottleReasons
  • AMD ROCm SMI library — power cap interface (rsmi_dev_power_cap_set)
  • RAPL (Running Average Power Limit) interface for Grace CPU power management

3. Agent Roles

Total agents: 80

Role Count Responsibilities
Lead Architect 2 Overall scheduling architecture decisions, cross-subsystem interface design, DR authorship for major decisions
Kernel Scheduling Researchers 8 CUDA/ROCm scheduler internals, MIG/MPS analysis, SM partitioning survey
Placement Researchers 6 Heterogeneous accelerator placement, topology modelling, workload profiling
Queueing Theory Specialists 5 DRF extensions for multi-resource GPU scheduling, fairness proof sketches
Gang Scheduling Specialists 5 Collective communication scheduling, all-or-nothing allocation protocols
Preemption Specialists 4 Checkpoint protocol design, preemption cost modelling
NUMA/Topology Analysts 4 NUMA topology modelling for Grace Superchip and SXM5 nodes
Energy Efficiency Researchers 4 Power capping, DVFS scheduling integration
Formal Specification Authors 8 TLA+ specification of scheduler state machine, safety and liveness invariants
Architects (sub-component) 10 Propose concrete scheduling algorithm designs for each domain
Internal Reviewers 8 Review research summaries and architecture proposals; cast green/yellow/red votes
Integration Liaisons 6 Interface with council-mem, council-net, council-telemetry, council-sec
Decision Record Authors 5 Author DRs for each resolved decision point; maintain UCXL provenance chain
Adversarial Critics 5 Challenge proposed designs; surface failure modes, starvation scenarios, livelock risks

Role distribution rationale: The large researcher cohort (37 agents covering 8 research domains) reflects the breadth of scheduling literature. The formal specification group (8 agents) is sized to produce parallel TLA+ modules. Internal reviewers and adversarial critics (13 agents combined) ensure no architecture proposal passes without rigorous challenge.

4. Key Deliverables

All artifacts are published to the DHT and addressable via UCXL.

4.1 Research Summaries

ucxl://council-sched:researcher@DistOS:scheduling/*^/research/gpu-kernel-scheduling.md
ucxl://council-sched:researcher@DistOS:scheduling/*^/research/heterogeneous-placement.md
ucxl://council-sched:researcher@DistOS:scheduling/*^/research/fair-queuing-drf.md
ucxl://council-sched:researcher@DistOS:scheduling/*^/research/gang-scheduling.md
ucxl://council-sched:researcher@DistOS:scheduling/*^/research/preemption-strategies.md
ucxl://council-sched:researcher@DistOS:scheduling/*^/research/numa-topology.md
ucxl://council-sched:researcher@DistOS:scheduling/*^/research/memory-oversubscription.md
ucxl://council-sched:researcher@DistOS:scheduling/*^/research/energy-aware-scheduling.md

4.2 Architecture Proposals

ucxl://council-sched:architect@DistOS:scheduling/*^/architecture/scheduler-overview.md
ucxl://council-sched:architect@DistOS:scheduling/*^/architecture/mig-mps-partitioning-model.md
ucxl://council-sched:architect@DistOS:scheduling/*^/architecture/placement-scoring-algorithm.md
ucxl://council-sched:architect@DistOS:scheduling/*^/architecture/drf-gpu-extension.md
ucxl://council-sched:architect@DistOS:scheduling/*^/architecture/gang-scheduler-protocol.md
ucxl://council-sched:architect@DistOS:scheduling/*^/architecture/preemption-protocol.md
ucxl://council-sched:architect@DistOS:scheduling/*^/architecture/energy-policy-interface.md

4.3 Decision Records

ucxl://council-sched:architect@DistOS:scheduling/*^/decisions/DR-SCHED-001-partition-model.md
ucxl://council-sched:architect@DistOS:scheduling/*^/decisions/DR-SCHED-002-placement-algorithm.md
ucxl://council-sched:architect@DistOS:scheduling/*^/decisions/DR-SCHED-003-queuing-policy.md
ucxl://council-sched:architect@DistOS:scheduling/*^/decisions/DR-SCHED-004-gang-protocol.md
ucxl://council-sched:architect@DistOS:scheduling/*^/decisions/DR-SCHED-005-preemption-model.md
ucxl://council-sched:architect@DistOS:scheduling/*^/decisions/DR-SCHED-006-energy-interface.md

4.4 Formal Specifications

ucxl://council-sched:verifier@DistOS:scheduling/*^/specs/SchedulerStateMachine.tla
ucxl://council-sched:verifier@DistOS:scheduling/*^/specs/GangScheduler.tla
ucxl://council-sched:verifier@DistOS:scheduling/*^/specs/PreemptionProtocol.tla
ucxl://council-sched:verifier@DistOS:scheduling/*^/specs/FairnessInvariants.tla

4.5 Interface Contracts (for other councils)

ucxl://council-sched:architect@DistOS:scheduling/*^/interfaces/sched-to-mem-contract.md
ucxl://council-sched:architect@DistOS:scheduling/*^/interfaces/sched-to-net-contract.md
ucxl://council-sched:architect@DistOS:scheduling/*^/interfaces/sched-to-telemetry-contract.md
ucxl://council-sched:architect@DistOS:scheduling/*^/interfaces/sched-to-sec-contract.md

5. Decision Points

The following are the major architectural questions council-sched must resolve. Each decision must produce a Decision Record with alternatives considered, evidence from research, and rationale for the chosen option.

DP-SCHED-001: Primary Partition Model

Question: Should the scheduler use MIG (hardware-enforced partition isolation), MPS (shared context with software-enforced limits), or a hybrid model where MIG is used for multi-tenant isolation and MPS for co-located jobs within a single tenant's partition?

Factors: isolation strength, context switch overhead, minimum allocation granularity, Blackwell support parity, operational complexity.

Dependency: Decision informs the security isolation model that council-sec will specify.

DP-SCHED-002: Placement Scoring Architecture

Question: Should placement be computed by a centralised scoring service (Borg-style) or a distributed, bid-based negotiation (Mesos offer model)? How does topology affinity (NVLink domain proximity) weight against fairness constraints?

Factors: convergence time at 1024-node scale, fault tolerance of the placement service itself, stale topology information handling, NVLink bandwidth utilisation efficiency.

DP-SCHED-003: Multi-Resource Fairness Extension

Question: DRF was designed for CPU/memory. GPU workloads introduce additional dominant resources: SM fraction, HBM3 bandwidth, NVLink bandwidth, and NVSwitch port occupancy. How many resource dimensions does the fairness model track, and what is the computational complexity of DRF at 80+ resource types?

Factors: implementation tractability, approximation error bounds, interaction with per-tenant quota enforcement.

DP-SCHED-004: Gang Scheduling Protocol

Question: Should gang scheduling use a two-phase reservation (reserve then commit) protocol, speculative allocation with rollback, or a backfill-with-hold strategy? What is the maximum tolerable scheduling delay for a 1024-GPU gang job?

Factors: cluster utilisation impact, deadlock risk in two-phase protocols, interaction with preemption.

DP-SCHED-005: Preemption Granularity

Question: Should preemption operate at kernel-granularity (CUDA stream checkpointing), job-granularity (full process checkpoint via CRIU-for-GPU), or a hybrid that allows kernel preemption for short jobs and process-level preemption for long jobs?

Factors: checkpoint latency (kernel-level: microseconds; process-level: seconds to tens of seconds for large model weights), HBM3 save/restore bandwidth cost, preemption frequency requirements.

DP-SCHED-006: Grace Superchip Scheduling Model

Question: For GH200 nodes with NVLink-C2C, the CPU and GPU share a unified virtual address space. Should the scheduler treat CPU and GPU on a Grace node as a single scheduling unit, or as two separate but affinity-linked resources? How does this interact with the memory model specified by council-mem?

Factors: utilisation efficiency, memory model consistency requirements, API ergonomics for user workloads, migration cost if job is split across a C2C boundary.

DP-SCHED-007: Energy Policy Interface

Question: Should energy-aware scheduling be a first-class scheduling objective (a weight in the placement score function), a hard constraint (power cap as a resource limit), or an advisory mechanism (scheduler receives power headroom hints and applies them at its discretion)?

Factors: SLO compatibility, predictability of power-capped execution, interaction with thermal management, coordination protocol with council-telemetry.

6. Dependencies

6.1 What council-sched Needs from Other Councils

Dependency Source Council Artifact Purpose
Memory pressure signals and eviction cost model council-mem ucxl://council-mem:architect@DistOS:memory/*^/interfaces/mem-to-sched-contract.md GPU memory oversubscription scheduling requires eviction cost estimates to avoid scheduling kernels that will immediately thrash
HBM3/DDR5 bandwidth allocation model council-mem ucxl://council-mem:architect@DistOS:memory/*^/architecture/bandwidth-allocation.md Bandwidth is a co-dominant scheduling resource; model must be consistent
Network topology map (NVLink domains, IB fabric) council-net ucxl://council-net:architect@DistOS:networking/*^/architecture/topology-model.md Topology-aware placement requires accurate NVLink domain membership and IB bisection bandwidth per rack
Network bandwidth reservation API council-net ucxl://council-net:architect@DistOS:networking/*^/interfaces/net-to-sched-contract.md Gang scheduling for allreduce jobs requires co-reserving network bandwidth
Resource metering API contract council-telemetry ucxl://council-telemetry:architect@DistOS:telemetry/*^/interfaces/telemetry-to-sched-contract.md Scheduler must feed placement and execution events to telemetry; power headroom signals flow back
Security isolation constraints council-sec ucxl://council-sec:architect@DistOS:security/*^/interfaces/sec-to-sched-contract.md Which tenants may share a GPU partition, MPS context constraints, minimum isolation requirements per tenant class

6.2 What Other Councils Need from council-sched

Consumer Council Artifact Required Purpose
council-mem Placement decisions and GPU assignment map Memory subsystem needs to know which GPU a job is placed on to configure HBM3 address space and NVLink fabric attachment
council-mem Preemption events and checkpoint triggers Memory tiering must snapshot GPU memory on preemption
council-net Job placement map with NVLink domain assignments Network subsystem configures NCCL topology files and RDMA QP mappings based on scheduler placement
council-telemetry Scheduling events stream (enqueue, dequeue, preempt, complete) Metering and cost attribution require per-job lifecycle events
council-sec Partition assignment per tenant Security subsystem enforces isolation based on scheduler-assigned partition IDs
council-verify TLA+ scheduler state machine spec Formal verification council model-checks scheduler invariants

7. WHOOSH Configuration

# WHOOSH council formation configuration for council-sched
council_id: council-sched
project: DistOS
subsystem: scheduling
gitea_label: chorus-entrypoint
gitea_repo: distos/scheduling

formation:
  target_agents: 80
  min_agents: 60
  wave:
    max_per_wave: 12
    min_per_wave: 6
    period_sec: 30
  placement:
    max_replicas_per_node: 2
  join_stagger_ms: 2000
  bootstrap_peers_min: 5

roles:
  - role: lead-architect
    count: 2
    model: claude-opus-4-6
    priority: high
  - role: researcher
    count: 37
    model: qwen2.5-coder:32b
    priority: normal
    subgroups:
      - tag: kernel-scheduling
        count: 8
      - tag: placement
        count: 6
      - tag: queuing-theory
        count: 5
      - tag: gang-scheduling
        count: 5
      - tag: preemption
        count: 4
      - tag: numa-topology
        count: 4
      - tag: energy
        count: 5
  - role: architect
    count: 10
    model: claude-opus-4-6
    priority: normal
  - role: verifier
    count: 8
    model: deepseek-coder-v2
    priority: normal
  - role: reviewer
    count: 8
    model: claude-opus-4-6
    priority: normal
  - role: integration-liaison
    count: 6
    model: qwen2.5-coder:32b
    priority: normal
  - role: decision-record-author
    count: 5
    model: claude-opus-4-6
    priority: normal
  - role: adversarial-critic
    count: 5
    model: claude-opus-4-6
    priority: normal

subchannels:
  - name: sched-research
    description: "Research discussion and literature synthesis"
    participants: [researcher, lead-architect]
    pubsub: true
  - name: sched-architecture
    description: "Architecture proposal discussion and voting"
    participants: [architect, lead-architect, reviewer, adversarial-critic]
    pubsub: true
  - name: sched-formal-spec
    description: "TLA+ specification authoring and review"
    participants: [verifier, lead-architect, reviewer]
    pubsub: false
  - name: sched-integration
    description: "Cross-council interface negotiation"
    participants: [integration-liaison, lead-architect]
    pubsub: false
  - name: sched-decisions
    description: "Decision record authoring and consensus"
    participants: [decision-record-author, lead-architect, reviewer]
    pubsub: true

quorum:
  # Architecture decisions require supermajority
  architecture_changes:
    policy: supermajority
    threshold: 0.667
    require_domain_role: true
    require_quality_role: true
    beat_minutes: 20
    timeout_beats: 6
  # Research summaries require simple majority
  research_summaries:
    policy: simple_majority
    threshold: 0.5
    require_domain_role: true
    require_quality_role: false
    beat_minutes: 15
    timeout_beats: 4
  # Formal specifications require supermajority with verifier sign-off
  formal_specs:
    policy: supermajority
    threshold: 0.667
    require_domain_role: true
    require_quality_role: true
    require_verifier: true
    beat_minutes: 25
    timeout_beats: 8
  # Cross-council interface contracts require unanimous lead-architect approval
  interface_contracts:
    policy: unanimous
    roles: [lead-architect, integration-liaison]
    beat_minutes: 30
    timeout_beats: 4

gates:
  kaching:
    p95_latency_ms: 250
    max_error_rate: 0.01
  backbeat:
    max_stream_lag: 200
  bootstrap:
    min_healthy_peers: 5
  join:
    min_success_rate: 0.80

review:
  beat_minutes: 20
  quorum:
    total_min: 3
    require_domain_role: true
    require_quality_role: true
  timeout_beats: 6
  no_self_approval: true

8. Success Criteria

  1. Research completeness: All 8 research domain summaries published to DHT with at least 5 primary references each, reviewed and approved by at least 3 council agents.

  2. Architecture coverage: Architectural proposals exist for all 7 major scheduling components (kernel dispatch, placement, queuing, gang, preemption, NUMA, energy). Each proposal addresses the 1024-node scale constraint explicitly.

  3. Decision records resolved: All 7 decision points (DP-SCHED-001 through DP-SCHED-007) have a corresponding Decision Record with at least 3 alternatives considered, evidence citations, and a chosen option ratified by council supermajority.

  4. Formal specifications: TLA+ specifications for the scheduler state machine, gang scheduling protocol, and preemption protocol. At least 2 of these must have model-checked safety invariants (no starvation for highest-priority jobs, gang deadlock freedom) verified by council-verify.

  5. Interface contracts ratified: All 4 interface contracts (to council-mem, council-net, council-telemetry, council-sec) are co-signed by integration liaisons from both councils.

  6. UCXL navigability: A human unfamiliar with the project should be able to navigate from any Decision Record to the research summary that motivated it using only UCXL temporal navigation, within 5 hops.

  7. Adversarial review pass: Each major architecture proposal has at minimum one adversarial critique documented and a resolution recorded. No proposal advances to formal specification with an unresolved red-vote critique.

9. Timeline

Phase 1: Research and Survey (Days 1-3)

Day 1:

  • WHOOSH forms council; all 80 agents join via wave deployment
  • Researchers self-assign to domain subgroups via sched-research subchannel
  • Literature survey begins: GPU kernel scheduling, placement, and fair queuing domains prioritised first (these are on the critical path for Decision Points DP-SCHED-001 through DP-SCHED-003)
  • Integration liaisons make initial contact with council-mem and council-net to understand their timelines

Day 2:

  • Remaining research domains surveyed: gang scheduling, preemption, NUMA, energy
  • Research summaries drafted in parallel across subgroups
  • First internal review cycle: reviewers read summaries and post green/yellow/red votes with rationale
  • Lead architects synthesise research findings into a preliminary scheduling design space map

Day 3:

  • Research summaries revised based on review feedback; final versions published to DHT
  • Adversarial critics challenge assumptions in each summary
  • Research phase gate: all 8 summaries must achieve simple majority approval before Phase 2 begins
  • Preliminary interface contract outlines shared with dependency councils

Phase 2: Architecture and Trade-offs (Days 3-6)

Day 3-4:

  • Architects propose concrete options for DP-SCHED-001 (partition model) and DP-SCHED-002 (placement scoring) — these are the highest-dependency decisions
  • Adversarial critics engage immediately; all alternatives documented
  • DP-SCHED-001 decision record drafted; council votes; DR published

Day 4-5:

  • Queuing model (DP-SCHED-003), gang scheduling (DP-SCHED-004), and preemption (DP-SCHED-005) design proposals concurrently authored
  • Inter-council synthesis session with council-mem to align on oversubscription and eviction signal interfaces
  • Inter-council synthesis session with council-net to align on topology model input format

Day 5-6:

  • Grace Superchip model (DP-SCHED-006) and energy interface (DP-SCHED-007) decisions resolved
  • All 7 Decision Records drafted; supermajority vote on each
  • Architecture overview document assembled from approved DRs
  • Architecture phase gate: all DPs resolved before Phase 3 begins

Phase 3: Formal Specification (Days 6-10)

Day 6-7:

  • Verifiers begin TLA+ specification of the scheduler state machine based on approved architecture
  • Architects continue refining component-level designs to resolve any ambiguities surfaced during spec authoring
  • council-verify engaged: share spec drafts for early model-checking feedback

Day 7-8:

  • Gang scheduler TLA+ module authored; parallel with preemption protocol spec
  • Fairness invariants formally stated: no starvation under bounded load, gang deadlock freedom, DRF monotonicity

Day 8-10:

  • Model checking runs submitted to council-verify
  • Counterexample analysis: any liveness or safety violations trigger architecture revision and updated DRs
  • Formal spec versions pinned in DHT; UCXL addresses published to dependent councils

Phase 4: Integration and Review (Days 10-12)

Day 10-11:

  • Interface contracts with council-mem, council-net, council-telemetry, council-sec finalised and submitted for co-signature
  • Cross-council integration session: scheduling placement decisions validated against network topology model
  • council-synth engaged for any unresolved conflicts with other councils' specifications

Day 11-12:

  • Final council review of complete scheduling specification
  • Adversarial critics run end-to-end failure scenario analysis
  • Any remaining yellow votes addressed with documented mitigations
  • Integration review gate: all interface contracts co-signed

Phase 5: Documentation and Narrative (Days 12-14)

Day 12-13:

  • Decision record authors produce narrative summaries of the scheduling architecture journey
  • council-docs receives the complete scheduling specification package for standardised formatting
  • UCXL navigability audit: spot-check 10 random decision paths for completeness

Day 14:

  • Final specification published
  • council-arch decision archaeology agents generate human-readable narrative of scheduling design evolution
  • Council formally dissolved; agents released back to WHOOSH pool
Reference in New Issue View Git Blame Copy Permalink