Initial DistOS project constitution and council design briefs

12 council design briefs for distributed OS specification project targeting 1024-node Hopper/Grace/Blackwell GPU cluster with Weka parallel filesystem. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 14:15:39 +11:00
commit 7f56ca4d46
13 changed files with 5857 additions and 0 deletions
--- a/councils/11-documentation.md
+++ b/councils/11-documentation.md
@@ -0,0 +1,399 @@
+# Council Design Brief: Documentation & Specification Formatting
+
+---
+
+## Council Identification
+
+| Field | Value |
+|-------|-------|
+| **Council ID** | `council-docs` |
+| **Mission** | Transform the raw technical outputs of all DistOS councils into a coherent, professionally formatted, cross-referenced, and accessible specification suite. Council-docs does not originate technical content — it is the editorial and production team responsible for ensuring that the collective intelligence of every council is rendered legible, navigable, and unambiguous to human readers. |
+| **UCXL Base Address** | `ucxl://council-docs:*@DistOS:docs/*^/` |
+| **Agent Count** | ~40 agents |
+| **Operates From** | Day 3 (continuous intake) through Day 14 (final publication) |
+
+---
+
+## Scope and Responsibilities
+
+Council-docs is the final production stage of the DistOS specification. Its scope is editorial, structural, and representational — not technical. Specifically:
+
+- Consuming outputs from all other councils via UCXL addressing and transforming them into polished specification documents
+- Enforcing consistent terminology, formatting, style, and cross-reference standards across all documents
+- Generating and maintaining the master glossary and terminology register for the DistOS project
+- Producing architecture decision records (ADRs) that capture the rationale for every major architectural choice, sourced from council decision artifacts
+- Creating visual representations (architecture diagrams, sequence diagrams, state machine diagrams, dependency maps) that complement written specifications
+- Managing document versioning as specifications evolve across the 14-day timeline
+- Maintaining bidirectional cross-references between all specification documents, ensuring that a change in one council's output is reflected in all dependent documents
+- Producing the final DistOS specification document suite as a coherent, publishable artifact
+
+Council-docs explicitly does **not**:
+- Make architectural or technical decisions
+- Resolve technical conflicts between councils (that is council-synth's domain)
+- Generate original technical analysis
+
+When a document review reveals technical ambiguity, inconsistency, or an apparent gap, council-docs raises a formal editorial query to the originating council and council-synth — it does not resolve the issue itself.
+
+---
+
+## Research Domains
+
+### 1. Standards Document Formatting (IEEE/ISO Style)
+
+The DistOS specification must be formatted to a standard that would be recognisable to engineers familiar with IEEE or ISO technical standards. This means:
+
+**IEEE 1016 (Software Design Descriptions)** — structure and content requirements for software design documentation. Council-docs will use IEEE 1016 as a structural template for the DistOS design specification.
+
+**ISO/IEC 26514 (Systems and software engineering — Requirements for designers and developers of user documentation)** — best practices for technical documentation clarity, completeness, and navigability.
+
+**RFC formatting conventions** — for protocol specifications, the IETF RFC format (with its formal requirement language: MUST, SHOULD, MAY, MUST NOT, SHOULD NOT as per RFC 2119) provides a well-understood convention for precise specification language.
+
+Key references:
+- IEEE Std 1016-2009. IEEE Standard for Information Technology — Systems Design — Software Design Descriptions.
+- ISO/IEC 26514:2008. Systems and software engineering — Requirements for designers and developers of user documentation.
+- Bradner, S. (1997). RFC 2119: Key words for use in RFCs to Indicate Requirement Levels. IETF.
+- Nygard, M. T. (2017). "Documenting Architecture Decisions." https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions
+
+### 2. Architecture Decision Records
+
+An ADR captures: the context that required a decision, the options considered, the decision made, and the consequences. For DistOS, every major architectural decision made by any council must be captured as an ADR.
+
+Council-docs is responsible for:
+- Extracting decision artifacts from UCXL addresses across all councils
+- Reformatting them as coherent, numbered ADRs (ADR-001, ADR-002, etc.)
+- Linking each ADR to the council-arch narratives that explain the decision chain
+- Maintaining an ADR status register (Proposed / Accepted / Deprecated / Superseded)
+
+**ADR format for DistOS:**
+
+```
+# ADR-NNN: [Title]
+
+## Status
+[Proposed | Accepted | Deprecated | Superseded by ADR-NNN]
+
+## Context
+[What situation required a decision? What forces were at play?]
+
+## Decision
+[What was decided?]
+
+## Consequences
+[What are the results of this decision? What becomes easier or harder?]
+
+## Source
+[UCXL addresses of the artifacts from which this ADR was derived]
+
+## Councils Involved
+[Which councils contributed to or were affected by this decision?]
+```
+
+Key references:
+- Nygard, M. T. (2011). "Documenting Architecture Decisions." Cognitect Blog.
+- Tyree, J., & Akerman, A. (2005). "Architecture Decisions: Demystifying Architecture." IEEE Software 22(2).
+- Richards, M., & Ford, N. (2020). Fundamentals of Software Architecture. O'Reilly Media. Chapter 19: Making Architecture Decisions.
+
+### 3. API Reference Documentation Generation
+
+For all APIs produced by council-api, council-docs will produce structured reference documentation following OpenAPI 3.x conventions where applicable, and custom structured formats for non-REST interfaces (GPU kernel APIs, consensus protocol messages, filesystem interfaces).
+
+Each API reference entry must include:
+- Endpoint or function signature with all parameters
+- Parameter types, constraints, and defaults
+- Pre-conditions and post-conditions (sourced from council-verify formal specs)
+- Error codes and their meaning
+- Example requests and responses
+- Security requirements
+- Rate limits and performance characteristics where specified
+
+**OpenAPI 3.1** — the current standard for REST API documentation. Council-docs will produce OpenAPI specs for any HTTP-based DistOS management interfaces.
+
+**Swagger/Redoc** — tooling for rendering OpenAPI specs into navigable HTML documentation.
+
+Key references:
+- OpenAPI Initiative. (2021). OpenAPI Specification v3.1.0. https://spec.openapis.org/oas/v3.1.0
+- Jacobson, D., Brail, G., & Woods, D. (2011). APIs: A Strategy Guide. O'Reilly Media.
+
+### 4. Diagram Generation Standards
+
+Technical diagrams must convey system structure and behaviour precisely and consistently. Council-docs Diagram Specialists will use:
+
+**C4 Model (Context, Container, Component, Code)** — for architecture overview diagrams at four levels of abstraction. Simon Brown's C4 model provides a structured hierarchy that prevents the common failure of a single "big ball of mud" architecture diagram.
+
+**UML Sequence Diagrams** — for inter-council and inter-component communication flows. Especially important for documenting GPU job submission flows, consensus protocol message exchanges, and checkpoint/recovery sequences.
+
+**UML State Machine Diagrams** — for component lifecycle documentation (GPU context states, scheduler job states, filesystem handle states).
+
+**Graphviz / Mermaid** — as the diagramming toolchain. Mermaid's text-based syntax integrates with Markdown documentation; Graphviz provides more precise layout control for complex dependency graphs.
+
+**Decision tree diagrams** — for documenting fault recovery decision logic, sourced from council-qa chaos playbooks and council-fault specifications.
+
+Key references:
+- Brown, S. (2018). The C4 Model for Visualising Software Architecture. Leanpub.
+- Fowler, M. (2003). UML Distilled: A Brief Guide to the Standard Object Modeling Language. Addison-Wesley.
+- OMG. (2017). Unified Modeling Language Specification Version 2.5.1.
+
+### 5. Cross-Reference Management
+
+In a specification suite produced by 10+ councils, cross-references between documents are as important as the documents themselves. A memory manager specification that references a scheduler guarantee must point to the specific version of that guarantee that was current when the memory spec was written.
+
+Council-docs Cross-Referencers will:
+- Maintain a master cross-reference registry mapping logical references to UCXL-versioned artifacts
+- Flag dangling references (references to artifacts that have been superseded or deleted)
+- Ensure that every inter-council dependency named in design briefs has a corresponding cross-reference in the final specification
+- Produce a dependency map showing which specification sections depend on which council outputs
+
+**UCXL temporal versioning** will be used to pin cross-references: when document A references a claim from document B, the reference includes the UCXL temporal address of the specific version of B that was current at the time of reference.
+
+Example:
+```
+The memory isolation guarantee stated in [MEM-ISOLATION-V3]
+(ucxl://council-mem:allocator@DistOS:mem/*~3/spec/isolation-guarantee)
+depends on the scheduler preemption guarantee in [SCHED-PREEMPT-V2]
+(ucxl://council-sched:scheduler@DistOS:sched/*~2/spec/preemption).
+```
+
+### 6. Terminology Standardisation
+
+Technical specifications fail when different sections use different words for the same concept, or the same word for different concepts. For a 1024-node GPU cluster OS, the terminology surface is large.
+
+Council-docs Terminology Guardians will maintain a canonical glossary that:
+- Defines every technical term used across all council outputs
+- Flags and resolves synonyms (is it "job", "task", "workload", or "kernel"?)
+- Maintains provenance for each definition (which council introduced it, when, and in what context)
+- Enforces consistent usage through editorial review
+
+The glossary is a living document; it is updated continuously as new terms emerge from council outputs.
+
+### 7. Accessibility and Readability of Technical Content
+
+The DistOS specification should be readable by:
+- GPU cluster architects who have not read the detailed formal specifications
+- Systems engineers implementing DistOS components
+- Future maintainers who need to understand why specific decisions were made
+- Auditors reviewing security and compliance properties
+
+Council-docs applies the following readability standards:
+- Flesch-Kincaid readability targets for narrative sections (technical precision does not require unreadable prose)
+- Consistent use of active voice in procedural sections
+- Progressive disclosure: overview sections before detail sections in every major document
+- Every document has a clearly stated purpose, audience, and scope in its opening section
+
+---
+
+## Agent Roles
+
+| Role | Count | Responsibilities |
+|------|-------|-----------------|
+| **Editors** | 15 | Review all council outputs for clarity, completeness, and internal consistency; rewrite for precision without altering technical content; raise editorial queries to originating councils |
+| **Formatters** | 10 | Enforce consistent document structure, heading hierarchy, numbering schemes, table formatting, and code block conventions across all outputs; apply IEEE/ISO and ADR templates |
+| **Cross-Referencers** | 5 | Maintain master cross-reference registry; validate all UCXL-pinned references; flag dangling or outdated references; produce dependency maps |
+| **Diagram Specialists** | 5 | Generate architecture diagrams (C4), sequence diagrams (UML), state machine diagrams, and decision trees from council outputs; maintain diagram source files in version control |
+| **Terminology Guardians** | 5 | Maintain master glossary; audit all council outputs for terminology consistency; propose canonical definitions; resolve synonym conflicts with originating councils |
+
+**Total: 40 agents**
+
+---
+
+## Key Deliverables
+
+| Deliverable | UCXL Address | Due Phase |
+|-------------|--------------|-----------|
+| Documentation Standards Guide | `ucxl://council-docs:formatter@DistOS:docs/*^/spec/documentation-standards` | Phase 1 |
+| Master Glossary (v0, seed terms) | `ucxl://council-docs:terminology-guardian@DistOS:docs/*^/glossary/master-v0` | Phase 2 |
+| ADR Template and Register | `ucxl://council-docs:formatter@DistOS:docs/*^/template/adr-template` | Phase 2 |
+| Cross-Reference Registry (live) | `ucxl://council-docs:cross-referencer@DistOS:docs/*^/registry/cross-references` | Phase 2 (continuous) |
+| Architecture Overview Document | `ucxl://council-docs:editor@DistOS:docs/*^/spec/architecture-overview` | Phase 3 |
+| C4 Architecture Diagrams (all levels) | `ucxl://council-docs:diagram-specialist@DistOS:docs/*^/diagram/c4-architecture` | Phase 3 |
+| API Reference Documentation | `ucxl://council-docs:editor@DistOS:docs/*^/reference/api-reference` | Phase 3 |
+| Subsystem Sequence Diagrams | `ucxl://council-docs:diagram-specialist@DistOS:docs/*^/diagram/sequence-diagrams` | Phase 3 |
+| ADR Compilation (all councils) | `ucxl://council-docs:formatter@DistOS:docs/*^/adr/complete-register` | Phase 4 |
+| Master Glossary (final) | `ucxl://council-docs:terminology-guardian@DistOS:docs/*^/glossary/master-final` | Phase 4 |
+| Formal Specification Suite (assembled) | `ucxl://council-docs:editor@DistOS:docs/*^/spec/formal-spec-suite` | Phase 5 |
+| State Machine Diagram Set | `ucxl://council-docs:diagram-specialist@DistOS:docs/*^/diagram/state-machines` | Phase 4 |
+| QA Test Report Formatted Edition | `ucxl://council-docs:formatter@DistOS:docs/*^/report/qa-formatted` | Phase 5 |
+| Final DistOS Specification Document | `ucxl://council-docs:editor@DistOS:docs/*^/spec/distos-specification-v1` | Phase 5 |
+
+---
+
+## Decision Points
+
+### DD-01: Primary Document Format Standard
+**Question:** What is the canonical format for the DistOS specification suite?
+
+**Options:**
+- A. Markdown with Mermaid diagrams — maximum portability, version-control friendly, renders on Gitea
+- B. LaTeX — maximum typographic control, produces publication-quality PDFs, standard for academic specifications
+- C. AsciiDoc — richer than Markdown, less complex than LaTeX, with strong tooling (Antora, Asciidoctor)
+- D. IEEE-formatted Word/ODT — directly compatible with standards submission processes
+
+**Implications:** Option A is most compatible with the CHORUS/UCXL toolchain and Gitea workflow. Option B produces the highest-quality PDF output but requires LaTeX toolchain expertise. The choice affects how cross-references and diagrams are managed throughout the project.
+
+### DD-02: ADR Ownership Model
+**Question:** Who owns the content of each ADR — the council that made the decision, or council-docs?
+
+**Options:**
+- A. Council-docs owns all ADRs; originating councils provide raw decision artifacts only
+- B. Originating councils own ADR content; council-docs applies formatting only
+- C. Co-ownership model: originating council writes ADR draft, council-docs editors revise for clarity, council-arch validates narrative accuracy
+
+**Implications:** Option C is highest quality but requires the most coordination overhead. The council-arch dependency is critical here: council-arch's UCXL traversal output is the primary source of decision chain data.
+
+### DD-03: Glossary Conflict Resolution Process
+**Question:** When two councils use the same term to mean different things, who resolves the conflict?
+
+**Options:**
+- A. Council-docs Terminology Guardians have final authority on terminology
+- B. The council that first defined the term retains its definition; later councils must use different terms
+- C. Terminology conflicts escalate to council-synth for resolution; council-docs documents the resolution
+
+**Implications:** Option C correctly positions council-synth as the cross-council arbitrator, but adds coordination overhead. Option A risks terminology decisions made by editors rather than domain experts.
+
+### DD-04: Diagram Fidelity vs. Speed Trade-off
+**Question:** For architecture diagrams, should council-docs produce high-fidelity manually-reviewed diagrams or automated/generated diagrams from machine-readable sources?
+
+**Options:**
+- A. All diagrams hand-crafted by Diagram Specialists for maximum clarity and accuracy
+- B. All diagrams auto-generated from structured UCXL metadata with manual review
+- C. Hybrid: C4 overview diagrams are hand-crafted; detailed sequence and state diagrams are auto-generated
+
+**Implications:** Option B produces diagrams that stay automatically consistent with the underlying specification as it changes. Option A produces higher-quality diagrams that may drift from the specification if not carefully maintained.
+
+### DD-05: Document Versioning Strategy
+**Question:** How are document versions managed as specifications evolve across the 14-day timeline?
+
+**Options:**
+- A. Semantic versioning (v0.1, v0.2 ... v1.0) with UCXL temporal addressing for history
+- B. Date-stamped snapshots only, with UCXL providing all history navigation
+- C. Git-based versioning with tags at each phase boundary, UCXL addresses pointing to git refs
+
+---
+
+## Dependencies on Other Councils
+
+| Council | Dependency Type | What Council-Docs Needs |
+|---------|----------------|------------------------|
+| **council-arch** | Critical upstream | Decision narratives and human-readable summaries to source ADR content; UCXL traversal paths for cross-reference validation |
+| **council-synth** | Critical upstream | Synthesis decisions (especially conflict resolutions) which form the spine of the architecture document |
+| **council-verify** | Upstream | Formal specification documents to edit and format; invariant definitions for the glossary |
+| **council-api** | Upstream | API definitions to transform into formatted API reference documentation |
+| **council-sched** | Upstream | Scheduler specification documents; sequence diagrams source data |
+| **council-mem** | Upstream | Memory manager specification documents; state machine source data |
+| **council-sec** | Upstream | Security model documents; threat model formatted edition |
+| **council-net** | Upstream | Network layer specification documents |
+| **council-fs** | Upstream | Filesystem integration specification documents |
+| **council-qa** | Upstream | QA test reports; defect register; chaos playbooks for inclusion in operational documentation |
+| **ALL councils** | Continuous | Terminology submissions for the master glossary; editorial query responses |
+
+**Special relationship with council-arch:** Council-docs depends on council-arch not just for content but for interpretive context. When editors encounter a decision that seems unmotivated or a specification section that lacks clear rationale, they query council-arch's narrative archive before raising a formal editorial query to the technical council. This prevents unnecessary interruptions to technical councils with questions that the archaeology record already answers.
+
+---
+
+## WHOOSH Configuration
+
+```yaml
+council: council-docs
+whoosh:
+  formation:
+    strategy: workflow-pipeline
+    # Documentation follows a pipeline: intake -> edit -> format -> cross-ref -> publish
+    stages:
+      - name: intake
+        roles: [editor]
+        trigger: new-artifact-available
+        description: Editors monitor UCXL for new artifacts from all councils
+      - name: edit
+        roles: [editor, terminology-guardian]
+        trigger: intake-complete
+        description: Editors revise for clarity; Terminology Guardians check glossary compliance
+      - name: format
+        roles: [formatter]
+        trigger: edit-complete
+        description: Formatters apply templates and structural standards
+      - name: cross-reference
+        roles: [cross-referencer]
+        trigger: format-complete
+        description: Cross-Referencers validate and register all inter-document references
+      - name: diagram
+        roles: [diagram-specialist]
+        trigger: parallel-with-format
+        description: Diagram Specialists generate/update visual representations
+      - name: publish
+        roles: [formatter, editor]
+        trigger: cross-reference-complete
+        description: Final document published to UCXL
+
+  quorum:
+    editorial-query: 2/5-editors    # Two editors must agree a query is warranted before sending
+    terminology-decision: 3/5-guardians-or-escalate  # Three guardians or escalate to council-synth
+    document-acceptance: editor + formatter + cross-referencer  # All three roles sign off per document
+
+  subchannels:
+    - id: artifact-intake
+      description: New artifacts arriving from all councils
+      subscribers: [council-docs-editors]
+      retention: full-history
+    - id: editorial-queries
+      description: Formal editorial queries sent to originating councils
+      subscribers: [all-councils, council-arch]
+      retention: full-history
+    - id: glossary-updates
+      description: New and revised glossary entries broadcast to all councils
+      subscribers: [all-councils]
+      retention: full-history
+    - id: document-published
+      description: Notification when a document section is published
+      subscribers: [all-councils, council-arch, council-synth]
+    - id: cross-reference-warnings
+      description: Dangling or stale cross-references requiring attention
+      subscribers: [council-synth, council-arch, originating-council]
+
+  continuous_monitoring:
+    # council-docs monitors all councils' UCXL streams for new artifacts
+    monitor_pattern: "ucxl://*:*@DistOS:*/*^/*"
+    trigger: on-new-artifact
+    action: queue-for-intake
+```
+
+---
+
+## Success Criteria
+
+Council-docs execution is successful when:
+
+1. **Complete coverage:** Every formal technical output from every other council has a corresponding edited, formatted, and cross-referenced document in the DistOS specification suite.
+
+2. **Terminology consistency:** The master glossary contains definitions for every distinct technical term used across all specification documents. Zero instances of unresolved synonym conflicts in published documents.
+
+3. **Cross-reference integrity:** All inter-document cross-references resolve to valid UCXL-addressed artifacts. Zero dangling references in the final published suite.
+
+4. **ADR completeness:** An ADR exists for every major architectural decision identified by council-arch's decision traversal. Every ADR links to its UCXL source artifacts.
+
+5. **Diagram coverage:** C4 context, container, and component diagrams exist for all major DistOS subsystems. Sequence diagrams exist for all major inter-component workflows. State machine diagrams exist for all major component lifecycles.
+
+6. **Readability standard:** A systems engineer with GPU cluster experience but no prior DistOS exposure can read the architecture overview document and correctly answer questions about the high-level design without consulting supplementary materials.
+
+7. **Format compliance:** All documents conform to the agreed formatting standard (IEEE-aligned Markdown or equivalent) without structural inconsistencies.
+
+8. **Version integrity:** Document versions are correctly pinned to UCXL temporal addresses. A reader can navigate to any historical version of any document by following the UCXL version chain.
+
+9. **Editorial queries resolved:** All formal editorial queries raised by council-docs have been responded to and resolved by the relevant technical councils.
+
+10. **Final specification published:** The complete DistOS Specification v1.0 document suite is published as a unified, navigable artifact by Day 14.
+
+---
+
+## Timeline Mapping
+
+| Phase | Days | Council-Docs Activities |
+|-------|------|------------------------|
+| **Phase 1: Research** | 1–3 | Establish documentation standards guide; define ADR template; seed master glossary with terms from DistOS design briefs; configure UCXL monitoring for artifact intake; coordinate with council-arch on narrative format integration |
+| **Phase 2: Architecture** | 3–6 | Begin editing first architecture outputs as they arrive from council-synth and subsystem councils; build cross-reference registry; produce ADRs from early architecture decisions; create initial C4 context diagram; first glossary broadcast to all councils |
+| **Phase 3: Formal Specification** | 6–10 | High-throughput editing and formatting as formal specs arrive from council-verify and all subsystem councils; API reference documentation production; sequence and state machine diagram generation; continuous cross-reference validation and glossary updates; ADR compilation accelerates |
+| **Phase 4: Integration** | 10–12 | Assemble subsystem documents into coherent specification suite; resolve all cross-reference issues; finalise ADR register; incorporate QA test report formatted edition; complete all diagrams; master glossary final review |
+| **Phase 5: Documentation** | 12–14 | Final editorial pass on all documents; publish DistOS Specification v1.0; produce publication-ready version of the complete specification suite; validate all UCXL addresses in published documents; confirm archaeology readability test with council-arch |
+
+---
+
+*Council Design Brief v1.0 — DistOS Project — council-docs*
+*Generated: 2026-02-24*