Files
CHORUS/docs/distos/councils/11-documentation.md

24 KiB
Raw Permalink Blame History

Council Design Brief: Documentation & Specification Formatting


Council Identification

Field Value
Council ID council-docs
Mission Transform the raw technical outputs of all DistOS councils into a coherent, professionally formatted, cross-referenced, and accessible specification suite. Council-docs does not originate technical content — it is the editorial and production team responsible for ensuring that the collective intelligence of every council is rendered legible, navigable, and unambiguous to human readers.
UCXL Base Address ucxl://council-docs:*@DistOS:docs/*^/
Agent Count ~40 agents
Operates From Day 3 (continuous intake) through Day 14 (final publication)

Scope and Responsibilities

Council-docs is the final production stage of the DistOS specification. Its scope is editorial, structural, and representational — not technical. Specifically:

  • Consuming outputs from all other councils via UCXL addressing and transforming them into polished specification documents
  • Enforcing consistent terminology, formatting, style, and cross-reference standards across all documents
  • Generating and maintaining the master glossary and terminology register for the DistOS project
  • Producing architecture decision records (ADRs) that capture the rationale for every major architectural choice, sourced from council decision artifacts
  • Creating visual representations (architecture diagrams, sequence diagrams, state machine diagrams, dependency maps) that complement written specifications
  • Managing document versioning as specifications evolve across the 14-day timeline
  • Maintaining bidirectional cross-references between all specification documents, ensuring that a change in one council's output is reflected in all dependent documents
  • Producing the final DistOS specification document suite as a coherent, publishable artifact

Council-docs explicitly does not:

  • Make architectural or technical decisions
  • Resolve technical conflicts between councils (that is council-synth's domain)
  • Generate original technical analysis

When a document review reveals technical ambiguity, inconsistency, or an apparent gap, council-docs raises a formal editorial query to the originating council and council-synth — it does not resolve the issue itself.


Research Domains

1. Standards Document Formatting (IEEE/ISO Style)

The DistOS specification must be formatted to a standard that would be recognisable to engineers familiar with IEEE or ISO technical standards. This means:

IEEE 1016 (Software Design Descriptions) — structure and content requirements for software design documentation. Council-docs will use IEEE 1016 as a structural template for the DistOS design specification.

ISO/IEC 26514 (Systems and software engineering — Requirements for designers and developers of user documentation) — best practices for technical documentation clarity, completeness, and navigability.

RFC formatting conventions — for protocol specifications, the IETF RFC format (with its formal requirement language: MUST, SHOULD, MAY, MUST NOT, SHOULD NOT as per RFC 2119) provides a well-understood convention for precise specification language.

Key references:

  • IEEE Std 1016-2009. IEEE Standard for Information Technology — Systems Design — Software Design Descriptions.
  • ISO/IEC 26514:2008. Systems and software engineering — Requirements for designers and developers of user documentation.
  • Bradner, S. (1997). RFC 2119: Key words for use in RFCs to Indicate Requirement Levels. IETF.
  • Nygard, M. T. (2017). "Documenting Architecture Decisions." https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions

2. Architecture Decision Records

An ADR captures: the context that required a decision, the options considered, the decision made, and the consequences. For DistOS, every major architectural decision made by any council must be captured as an ADR.

Council-docs is responsible for:

  • Extracting decision artifacts from UCXL addresses across all councils
  • Reformatting them as coherent, numbered ADRs (ADR-001, ADR-002, etc.)
  • Linking each ADR to the council-arch narratives that explain the decision chain
  • Maintaining an ADR status register (Proposed / Accepted / Deprecated / Superseded)

ADR format for DistOS:

# ADR-NNN: [Title]

## Status
[Proposed | Accepted | Deprecated | Superseded by ADR-NNN]

## Context
[What situation required a decision? What forces were at play?]

## Decision
[What was decided?]

## Consequences
[What are the results of this decision? What becomes easier or harder?]

## Source
[UCXL addresses of the artifacts from which this ADR was derived]

## Councils Involved
[Which councils contributed to or were affected by this decision?]

Key references:

  • Nygard, M. T. (2011). "Documenting Architecture Decisions." Cognitect Blog.
  • Tyree, J., & Akerman, A. (2005). "Architecture Decisions: Demystifying Architecture." IEEE Software 22(2).
  • Richards, M., & Ford, N. (2020). Fundamentals of Software Architecture. O'Reilly Media. Chapter 19: Making Architecture Decisions.

3. API Reference Documentation Generation

For all APIs produced by council-api, council-docs will produce structured reference documentation following OpenAPI 3.x conventions where applicable, and custom structured formats for non-REST interfaces (GPU kernel APIs, consensus protocol messages, filesystem interfaces).

Each API reference entry must include:

  • Endpoint or function signature with all parameters
  • Parameter types, constraints, and defaults
  • Pre-conditions and post-conditions (sourced from council-verify formal specs)
  • Error codes and their meaning
  • Example requests and responses
  • Security requirements
  • Rate limits and performance characteristics where specified

OpenAPI 3.1 — the current standard for REST API documentation. Council-docs will produce OpenAPI specs for any HTTP-based DistOS management interfaces.

Swagger/Redoc — tooling for rendering OpenAPI specs into navigable HTML documentation.

Key references:

  • OpenAPI Initiative. (2021). OpenAPI Specification v3.1.0. https://spec.openapis.org/oas/v3.1.0
  • Jacobson, D., Brail, G., & Woods, D. (2011). APIs: A Strategy Guide. O'Reilly Media.

4. Diagram Generation Standards

Technical diagrams must convey system structure and behaviour precisely and consistently. Council-docs Diagram Specialists will use:

C4 Model (Context, Container, Component, Code) — for architecture overview diagrams at four levels of abstraction. Simon Brown's C4 model provides a structured hierarchy that prevents the common failure of a single "big ball of mud" architecture diagram.

UML Sequence Diagrams — for inter-council and inter-component communication flows. Especially important for documenting GPU job submission flows, consensus protocol message exchanges, and checkpoint/recovery sequences.

UML State Machine Diagrams — for component lifecycle documentation (GPU context states, scheduler job states, filesystem handle states).

Graphviz / Mermaid — as the diagramming toolchain. Mermaid's text-based syntax integrates with Markdown documentation; Graphviz provides more precise layout control for complex dependency graphs.

Decision tree diagrams — for documenting fault recovery decision logic, sourced from council-qa chaos playbooks and council-fault specifications.

Key references:

  • Brown, S. (2018). The C4 Model for Visualising Software Architecture. Leanpub.
  • Fowler, M. (2003). UML Distilled: A Brief Guide to the Standard Object Modeling Language. Addison-Wesley.
  • OMG. (2017). Unified Modeling Language Specification Version 2.5.1.

5. Cross-Reference Management

In a specification suite produced by 10+ councils, cross-references between documents are as important as the documents themselves. A memory manager specification that references a scheduler guarantee must point to the specific version of that guarantee that was current when the memory spec was written.

Council-docs Cross-Referencers will:

  • Maintain a master cross-reference registry mapping logical references to UCXL-versioned artifacts
  • Flag dangling references (references to artifacts that have been superseded or deleted)
  • Ensure that every inter-council dependency named in design briefs has a corresponding cross-reference in the final specification
  • Produce a dependency map showing which specification sections depend on which council outputs

UCXL temporal versioning will be used to pin cross-references: when document A references a claim from document B, the reference includes the UCXL temporal address of the specific version of B that was current at the time of reference.

Example:

The memory isolation guarantee stated in [MEM-ISOLATION-V3]
(ucxl://council-mem:allocator@DistOS:mem/*~3/spec/isolation-guarantee)
depends on the scheduler preemption guarantee in [SCHED-PREEMPT-V2]
(ucxl://council-sched:scheduler@DistOS:sched/*~2/spec/preemption).

6. Terminology Standardisation

Technical specifications fail when different sections use different words for the same concept, or the same word for different concepts. For a 1024-node GPU cluster OS, the terminology surface is large.

Council-docs Terminology Guardians will maintain a canonical glossary that:

  • Defines every technical term used across all council outputs
  • Flags and resolves synonyms (is it "job", "task", "workload", or "kernel"?)
  • Maintains provenance for each definition (which council introduced it, when, and in what context)
  • Enforces consistent usage through editorial review

The glossary is a living document; it is updated continuously as new terms emerge from council outputs.

7. Accessibility and Readability of Technical Content

The DistOS specification should be readable by:

  • GPU cluster architects who have not read the detailed formal specifications
  • Systems engineers implementing DistOS components
  • Future maintainers who need to understand why specific decisions were made
  • Auditors reviewing security and compliance properties

Council-docs applies the following readability standards:

  • Flesch-Kincaid readability targets for narrative sections (technical precision does not require unreadable prose)
  • Consistent use of active voice in procedural sections
  • Progressive disclosure: overview sections before detail sections in every major document
  • Every document has a clearly stated purpose, audience, and scope in its opening section

Agent Roles

Role Count Responsibilities
Editors 15 Review all council outputs for clarity, completeness, and internal consistency; rewrite for precision without altering technical content; raise editorial queries to originating councils
Formatters 10 Enforce consistent document structure, heading hierarchy, numbering schemes, table formatting, and code block conventions across all outputs; apply IEEE/ISO and ADR templates
Cross-Referencers 5 Maintain master cross-reference registry; validate all UCXL-pinned references; flag dangling or outdated references; produce dependency maps
Diagram Specialists 5 Generate architecture diagrams (C4), sequence diagrams (UML), state machine diagrams, and decision trees from council outputs; maintain diagram source files in version control
Terminology Guardians 5 Maintain master glossary; audit all council outputs for terminology consistency; propose canonical definitions; resolve synonym conflicts with originating councils

Total: 40 agents


Key Deliverables

Deliverable UCXL Address Due Phase
Documentation Standards Guide ucxl://council-docs:formatter@DistOS:docs/*^/spec/documentation-standards Phase 1
Master Glossary (v0, seed terms) ucxl://council-docs:terminology-guardian@DistOS:docs/*^/glossary/master-v0 Phase 2
ADR Template and Register ucxl://council-docs:formatter@DistOS:docs/*^/template/adr-template Phase 2
Cross-Reference Registry (live) ucxl://council-docs:cross-referencer@DistOS:docs/*^/registry/cross-references Phase 2 (continuous)
Architecture Overview Document ucxl://council-docs:editor@DistOS:docs/*^/spec/architecture-overview Phase 3
C4 Architecture Diagrams (all levels) ucxl://council-docs:diagram-specialist@DistOS:docs/*^/diagram/c4-architecture Phase 3
API Reference Documentation ucxl://council-docs:editor@DistOS:docs/*^/reference/api-reference Phase 3
Subsystem Sequence Diagrams ucxl://council-docs:diagram-specialist@DistOS:docs/*^/diagram/sequence-diagrams Phase 3
ADR Compilation (all councils) ucxl://council-docs:formatter@DistOS:docs/*^/adr/complete-register Phase 4
Master Glossary (final) ucxl://council-docs:terminology-guardian@DistOS:docs/*^/glossary/master-final Phase 4
Formal Specification Suite (assembled) ucxl://council-docs:editor@DistOS:docs/*^/spec/formal-spec-suite Phase 5
State Machine Diagram Set ucxl://council-docs:diagram-specialist@DistOS:docs/*^/diagram/state-machines Phase 4
QA Test Report Formatted Edition ucxl://council-docs:formatter@DistOS:docs/*^/report/qa-formatted Phase 5
Final DistOS Specification Document ucxl://council-docs:editor@DistOS:docs/*^/spec/distos-specification-v1 Phase 5

Decision Points

DD-01: Primary Document Format Standard

Question: What is the canonical format for the DistOS specification suite?

Options:

  • A. Markdown with Mermaid diagrams — maximum portability, version-control friendly, renders on Gitea
  • B. LaTeX — maximum typographic control, produces publication-quality PDFs, standard for academic specifications
  • C. AsciiDoc — richer than Markdown, less complex than LaTeX, with strong tooling (Antora, Asciidoctor)
  • D. IEEE-formatted Word/ODT — directly compatible with standards submission processes

Implications: Option A is most compatible with the CHORUS/UCXL toolchain and Gitea workflow. Option B produces the highest-quality PDF output but requires LaTeX toolchain expertise. The choice affects how cross-references and diagrams are managed throughout the project.

DD-02: ADR Ownership Model

Question: Who owns the content of each ADR — the council that made the decision, or council-docs?

Options:

  • A. Council-docs owns all ADRs; originating councils provide raw decision artifacts only
  • B. Originating councils own ADR content; council-docs applies formatting only
  • C. Co-ownership model: originating council writes ADR draft, council-docs editors revise for clarity, council-arch validates narrative accuracy

Implications: Option C is highest quality but requires the most coordination overhead. The council-arch dependency is critical here: council-arch's UCXL traversal output is the primary source of decision chain data.

DD-03: Glossary Conflict Resolution Process

Question: When two councils use the same term to mean different things, who resolves the conflict?

Options:

  • A. Council-docs Terminology Guardians have final authority on terminology
  • B. The council that first defined the term retains its definition; later councils must use different terms
  • C. Terminology conflicts escalate to council-synth for resolution; council-docs documents the resolution

Implications: Option C correctly positions council-synth as the cross-council arbitrator, but adds coordination overhead. Option A risks terminology decisions made by editors rather than domain experts.

DD-04: Diagram Fidelity vs. Speed Trade-off

Question: For architecture diagrams, should council-docs produce high-fidelity manually-reviewed diagrams or automated/generated diagrams from machine-readable sources?

Options:

  • A. All diagrams hand-crafted by Diagram Specialists for maximum clarity and accuracy
  • B. All diagrams auto-generated from structured UCXL metadata with manual review
  • C. Hybrid: C4 overview diagrams are hand-crafted; detailed sequence and state diagrams are auto-generated

Implications: Option B produces diagrams that stay automatically consistent with the underlying specification as it changes. Option A produces higher-quality diagrams that may drift from the specification if not carefully maintained.

DD-05: Document Versioning Strategy

Question: How are document versions managed as specifications evolve across the 14-day timeline?

Options:

  • A. Semantic versioning (v0.1, v0.2 ... v1.0) with UCXL temporal addressing for history
  • B. Date-stamped snapshots only, with UCXL providing all history navigation
  • C. Git-based versioning with tags at each phase boundary, UCXL addresses pointing to git refs

Dependencies on Other Councils

Council Dependency Type What Council-Docs Needs
council-arch Critical upstream Decision narratives and human-readable summaries to source ADR content; UCXL traversal paths for cross-reference validation
council-synth Critical upstream Synthesis decisions (especially conflict resolutions) which form the spine of the architecture document
council-verify Upstream Formal specification documents to edit and format; invariant definitions for the glossary
council-api Upstream API definitions to transform into formatted API reference documentation
council-sched Upstream Scheduler specification documents; sequence diagrams source data
council-mem Upstream Memory manager specification documents; state machine source data
council-sec Upstream Security model documents; threat model formatted edition
council-net Upstream Network layer specification documents
council-fs Upstream Filesystem integration specification documents
council-qa Upstream QA test reports; defect register; chaos playbooks for inclusion in operational documentation
ALL councils Continuous Terminology submissions for the master glossary; editorial query responses

Special relationship with council-arch: Council-docs depends on council-arch not just for content but for interpretive context. When editors encounter a decision that seems unmotivated or a specification section that lacks clear rationale, they query council-arch's narrative archive before raising a formal editorial query to the technical council. This prevents unnecessary interruptions to technical councils with questions that the archaeology record already answers.


WHOOSH Configuration

council: council-docs
whoosh:
  formation:
    strategy: workflow-pipeline
    # Documentation follows a pipeline: intake -> edit -> format -> cross-ref -> publish
    stages:
      - name: intake
        roles: [editor]
        trigger: new-artifact-available
        description: Editors monitor UCXL for new artifacts from all councils
      - name: edit
        roles: [editor, terminology-guardian]
        trigger: intake-complete
        description: Editors revise for clarity; Terminology Guardians check glossary compliance
      - name: format
        roles: [formatter]
        trigger: edit-complete
        description: Formatters apply templates and structural standards
      - name: cross-reference
        roles: [cross-referencer]
        trigger: format-complete
        description: Cross-Referencers validate and register all inter-document references
      - name: diagram
        roles: [diagram-specialist]
        trigger: parallel-with-format
        description: Diagram Specialists generate/update visual representations
      - name: publish
        roles: [formatter, editor]
        trigger: cross-reference-complete
        description: Final document published to UCXL

  quorum:
    editorial-query: 2/5-editors    # Two editors must agree a query is warranted before sending
    terminology-decision: 3/5-guardians-or-escalate  # Three guardians or escalate to council-synth
    document-acceptance: editor + formatter + cross-referencer  # All three roles sign off per document

  subchannels:
    - id: artifact-intake
      description: New artifacts arriving from all councils
      subscribers: [council-docs-editors]
      retention: full-history
    - id: editorial-queries
      description: Formal editorial queries sent to originating councils
      subscribers: [all-councils, council-arch]
      retention: full-history
    - id: glossary-updates
      description: New and revised glossary entries broadcast to all councils
      subscribers: [all-councils]
      retention: full-history
    - id: document-published
      description: Notification when a document section is published
      subscribers: [all-councils, council-arch, council-synth]
    - id: cross-reference-warnings
      description: Dangling or stale cross-references requiring attention
      subscribers: [council-synth, council-arch, originating-council]

  continuous_monitoring:
    # council-docs monitors all councils' UCXL streams for new artifacts
    monitor_pattern: "ucxl://*:*@DistOS:*/*^/*"
    trigger: on-new-artifact
    action: queue-for-intake

Success Criteria

Council-docs execution is successful when:

  1. Complete coverage: Every formal technical output from every other council has a corresponding edited, formatted, and cross-referenced document in the DistOS specification suite.

  2. Terminology consistency: The master glossary contains definitions for every distinct technical term used across all specification documents. Zero instances of unresolved synonym conflicts in published documents.

  3. Cross-reference integrity: All inter-document cross-references resolve to valid UCXL-addressed artifacts. Zero dangling references in the final published suite.

  4. ADR completeness: An ADR exists for every major architectural decision identified by council-arch's decision traversal. Every ADR links to its UCXL source artifacts.

  5. Diagram coverage: C4 context, container, and component diagrams exist for all major DistOS subsystems. Sequence diagrams exist for all major inter-component workflows. State machine diagrams exist for all major component lifecycles.

  6. Readability standard: A systems engineer with GPU cluster experience but no prior DistOS exposure can read the architecture overview document and correctly answer questions about the high-level design without consulting supplementary materials.

  7. Format compliance: All documents conform to the agreed formatting standard (IEEE-aligned Markdown or equivalent) without structural inconsistencies.

  8. Version integrity: Document versions are correctly pinned to UCXL temporal addresses. A reader can navigate to any historical version of any document by following the UCXL version chain.

  9. Editorial queries resolved: All formal editorial queries raised by council-docs have been responded to and resolved by the relevant technical councils.

  10. Final specification published: The complete DistOS Specification v1.0 document suite is published as a unified, navigable artifact by Day 14.


Timeline Mapping

Phase Days Council-Docs Activities
Phase 1: Research 13 Establish documentation standards guide; define ADR template; seed master glossary with terms from DistOS design briefs; configure UCXL monitoring for artifact intake; coordinate with council-arch on narrative format integration
Phase 2: Architecture 36 Begin editing first architecture outputs as they arrive from council-synth and subsystem councils; build cross-reference registry; produce ADRs from early architecture decisions; create initial C4 context diagram; first glossary broadcast to all councils
Phase 3: Formal Specification 610 High-throughput editing and formatting as formal specs arrive from council-verify and all subsystem councils; API reference documentation production; sequence and state machine diagram generation; continuous cross-reference validation and glossary updates; ADR compilation accelerates
Phase 4: Integration 1012 Assemble subsystem documents into coherent specification suite; resolve all cross-reference issues; finalise ADR register; incorporate QA test report formatted edition; complete all diagrams; master glossary final review
Phase 5: Documentation 1214 Final editorial pass on all documents; publish DistOS Specification v1.0; produce publication-ready version of the complete specification suite; validate all UCXL addresses in published documents; confirm archaeology readability test with council-arch

Council Design Brief v1.0 — DistOS Project — council-docs Generated: 2026-02-24