Phase 2: Implement Execution Environment Abstraction (v0.3.0)

This commit implements Phase 2 of the CHORUS Task Execution Engine development plan,
providing a comprehensive execution environment abstraction layer with Docker
container sandboxing support.

## New Features

### Core Sandbox Interface
- Comprehensive ExecutionSandbox interface with isolated task execution
- Support for command execution, file I/O, environment management
- Resource usage monitoring and sandbox lifecycle management
- Standardized error handling with SandboxError types and categories

### Docker Container Sandbox Implementation
- Full Docker API integration with secure container creation
- Transparent repository mounting with configurable read/write access
- Advanced security policies with capability dropping and privilege controls
- Comprehensive resource limits (CPU, memory, disk, processes, file handles)
- Support for tmpfs mounts, masked paths, and read-only bind mounts
- Container lifecycle management with proper cleanup and health monitoring

### Security & Resource Management
- Configurable security policies with SELinux, AppArmor, and Seccomp support
- Fine-grained capability management with secure defaults
- Network isolation options with configurable DNS and proxy settings
- Resource monitoring with real-time CPU, memory, and network usage tracking
- Comprehensive ulimits configuration for process and file handle limits

### Repository Integration
- Seamless repository mounting from local paths to container workspaces
- Git configuration support with user credentials and global settings
- File inclusion/exclusion patterns for selective repository access
- Configurable permissions and ownership for mounted repositories

### Testing Infrastructure
- Comprehensive test suite with 60+ test cases covering all functionality
- Docker integration tests with Alpine Linux containers (skipped in short mode)
- Mock sandbox implementation for unit testing without Docker dependencies
- Security policy validation tests with read-only filesystem enforcement
- Resource usage monitoring and cleanup verification tests

## Technical Details

### Dependencies Added
- github.com/docker/docker v28.4.0+incompatible - Docker API client
- github.com/docker/go-connections v0.6.0 - Docker connection utilities
- github.com/docker/go-units v0.5.0 - Docker units and formatting
- Associated Docker API dependencies for complete container management

### Architecture
- Interface-driven design enabling multiple sandbox implementations
- Comprehensive configuration structures for all sandbox aspects
- Resource usage tracking with detailed metrics collection
- Error handling with retryable error classification
- Proper cleanup and resource management throughout sandbox lifecycle

### Compatibility
- Maintains backward compatibility with existing CHORUS architecture
- Designed for future integration with Phase 3 Core Task Execution Engine
- Extensible design supporting additional sandbox implementations (VM, process)

This Phase 2 implementation provides the foundation for secure, isolated task
execution that will be integrated with the AI model providers from Phase 1
in the upcoming Phase 3 development.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
anthonyrawlins
2025-09-25 14:28:08 +10:00
parent d1252ade69
commit 8d9b62daf3
653 changed files with 88039 additions and 3766 deletions

View File

@@ -1,16 +1,5 @@
# Copyright The OpenTelemetry Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# SPDX-License-Identifier: Apache-2.0
TOOLS_MOD_DIR := ./internal/tools
@@ -22,11 +11,15 @@ ALL_COVERAGE_MOD_DIRS := $(shell find . -type f -name 'go.mod' -exec dirname {}
GO = go
TIMEOUT = 60
# User to run as in docker images.
DOCKER_USER=$(shell id -u):$(shell id -g)
DEPENDENCIES_DOCKERFILE=./dependencies.Dockerfile
.DEFAULT_GOAL := precommit
.PHONY: precommit ci
precommit: generate dependabot-generate license-check vanity-import-fix misspell go-mod-tidy golangci-lint-fix test-default
ci: generate dependabot-check license-check lint vanity-import-check build test-default check-clean-work-tree test-coverage
precommit: generate toolchain-check license-check misspell go-mod-tidy golangci-lint-fix verify-readmes verify-mods test-default
ci: generate toolchain-check license-check lint vanity-import-check verify-readmes verify-mods build test-default check-clean-work-tree test-coverage
# Tools
@@ -34,27 +27,24 @@ TOOLS = $(CURDIR)/.tools
$(TOOLS):
@mkdir -p $@
$(TOOLS)/%: | $(TOOLS)
$(TOOLS)/%: $(TOOLS_MOD_DIR)/go.mod | $(TOOLS)
cd $(TOOLS_MOD_DIR) && \
$(GO) build -o $@ $(PACKAGE)
MULTIMOD = $(TOOLS)/multimod
$(TOOLS)/multimod: PACKAGE=go.opentelemetry.io/build-tools/multimod
SEMCONVGEN = $(TOOLS)/semconvgen
$(TOOLS)/semconvgen: PACKAGE=go.opentelemetry.io/build-tools/semconvgen
CROSSLINK = $(TOOLS)/crosslink
$(TOOLS)/crosslink: PACKAGE=go.opentelemetry.io/build-tools/crosslink
SEMCONVKIT = $(TOOLS)/semconvkit
$(TOOLS)/semconvkit: PACKAGE=go.opentelemetry.io/otel/$(TOOLS_MOD_DIR)/semconvkit
DBOTCONF = $(TOOLS)/dbotconf
$(TOOLS)/dbotconf: PACKAGE=go.opentelemetry.io/build-tools/dbotconf
VERIFYREADMES = $(TOOLS)/verifyreadmes
$(TOOLS)/verifyreadmes: PACKAGE=go.opentelemetry.io/otel/$(TOOLS_MOD_DIR)/verifyreadmes
GOLANGCI_LINT = $(TOOLS)/golangci-lint
$(TOOLS)/golangci-lint: PACKAGE=github.com/golangci/golangci-lint/cmd/golangci-lint
$(TOOLS)/golangci-lint: PACKAGE=github.com/golangci/golangci-lint/v2/cmd/golangci-lint
MISSPELL = $(TOOLS)/misspell
$(TOOLS)/misspell: PACKAGE=github.com/client9/misspell/cmd/misspell
@@ -68,11 +58,17 @@ $(TOOLS)/stringer: PACKAGE=golang.org/x/tools/cmd/stringer
PORTO = $(TOOLS)/porto
$(TOOLS)/porto: PACKAGE=github.com/jcchavezs/porto/cmd/porto
GOJQ = $(TOOLS)/gojq
$(TOOLS)/gojq: PACKAGE=github.com/itchyny/gojq/cmd/gojq
GOTMPL = $(TOOLS)/gotmpl
$(GOTMPL): PACKAGE=go.opentelemetry.io/build-tools/gotmpl
GORELEASE = $(TOOLS)/gorelease
$(GORELEASE): PACKAGE=golang.org/x/exp/cmd/gorelease
GOVULNCHECK = $(TOOLS)/govulncheck
$(TOOLS)/govulncheck: PACKAGE=golang.org/x/vuln/cmd/govulncheck
.PHONY: tools
tools: $(CROSSLINK) $(DBOTCONF) $(GOLANGCI_LINT) $(MISSPELL) $(GOCOVMERGE) $(STRINGER) $(PORTO) $(GOJQ) $(SEMCONVGEN) $(MULTIMOD) $(SEMCONVKIT)
tools: $(CROSSLINK) $(GOLANGCI_LINT) $(MISSPELL) $(GOCOVMERGE) $(STRINGER) $(PORTO) $(VERIFYREADMES) $(MULTIMOD) $(SEMCONVKIT) $(GOTMPL) $(GORELEASE)
# Virtualized python tools via docker
@@ -89,20 +85,20 @@ PIP := $(PYTOOLS)/pip
WORKDIR := /workdir
# The python image to use for the virtual environment.
PYTHONIMAGE := python:3.11.3-slim-bullseye
PYTHONIMAGE := $(shell awk '$$4=="python" {print $$2}' $(DEPENDENCIES_DOCKERFILE))
# Run the python image with the current directory mounted.
DOCKERPY := docker run --rm -v "$(CURDIR):$(WORKDIR)" -w $(WORKDIR) $(PYTHONIMAGE)
DOCKERPY := docker run --rm -u $(DOCKER_USER) -v "$(CURDIR):$(WORKDIR)" -w $(WORKDIR) $(PYTHONIMAGE)
# Create a virtual environment for Python tools.
$(PYTOOLS):
# The `--upgrade` flag is needed to ensure that the virtual environment is
# created with the latest pip version.
@$(DOCKERPY) bash -c "python3 -m venv $(VENVDIR) && $(PIP) install --upgrade pip"
@$(DOCKERPY) bash -c "python3 -m venv $(VENVDIR) && $(PIP) install --upgrade --cache-dir=$(WORKDIR)/.cache/pip pip"
# Install python packages into the virtual environment.
$(PYTOOLS)/%: | $(PYTOOLS)
@$(DOCKERPY) $(PIP) install -r requirements.txt
$(PYTOOLS)/%: $(PYTOOLS)
@$(DOCKERPY) $(PIP) install --cache-dir=$(WORKDIR)/.cache/pip -r requirements.txt
CODESPELL = $(PYTOOLS)/codespell
$(CODESPELL): PACKAGE=codespell
@@ -110,13 +106,24 @@ $(CODESPELL): PACKAGE=codespell
# Generate
.PHONY: generate
generate: go-generate vanity-import-fix
generate: $(OTEL_GO_MOD_DIRS:%=generate/%)
generate/%: DIR=$*
generate/%: | $(STRINGER) $(PORTO)
.PHONY: go-generate
go-generate: $(OTEL_GO_MOD_DIRS:%=go-generate/%)
go-generate/%: DIR=$*
go-generate/%: $(STRINGER) $(GOTMPL)
@echo "$(GO) generate $(DIR)/..." \
&& cd $(DIR) \
&& PATH="$(TOOLS):$${PATH}" $(GO) generate ./... && $(PORTO) -w .
&& PATH="$(TOOLS):$${PATH}" $(GO) generate ./...
.PHONY: vanity-import-fix
vanity-import-fix: $(PORTO)
@$(PORTO) --include-internal -w .
# Generate go.work file for local development.
.PHONY: go-work
go-work: $(CROSSLINK)
$(CROSSLINK) work --root=$(shell pwd) --go=1.22.7
# Build
@@ -139,12 +146,14 @@ build-tests/%:
# Tests
TEST_TARGETS := test-default test-bench test-short test-verbose test-race
TEST_TARGETS := test-default test-bench test-short test-verbose test-race test-concurrent-safe
.PHONY: $(TEST_TARGETS) test
test-default test-race: ARGS=-race
test-bench: ARGS=-run=xxxxxMatchNothingxxxxx -test.benchtime=1ms -bench=.
test-short: ARGS=-short
test-verbose: ARGS=-v -race
test-concurrent-safe: ARGS=-run=ConcurrentSafe -count=100 -race
test-concurrent-safe: TIMEOUT=120
$(TEST_TARGETS): test
test: $(OTEL_GO_MOD_DIRS:%=test/%)
test/%: DIR=$*
@@ -158,7 +167,7 @@ test/%:
COVERAGE_MODE = atomic
COVERAGE_PROFILE = coverage.out
.PHONY: test-coverage
test-coverage: | $(GOCOVMERGE)
test-coverage: $(GOCOVMERGE)
@set -e; \
printf "" > coverage.txt; \
for dir in $(ALL_COVERAGE_MOD_DIRS); do \
@@ -172,70 +181,81 @@ test-coverage: | $(GOCOVMERGE)
done; \
$(GOCOVMERGE) $$(find . -name coverage.out) > coverage.txt
.PHONY: benchmark
benchmark: $(OTEL_GO_MOD_DIRS:%=benchmark/%)
benchmark/%:
@echo "$(GO) test -run=xxxxxMatchNothingxxxxx -bench=. $*..." \
&& cd $* \
&& $(GO) list ./... \
| grep -v third_party \
| xargs $(GO) test -run=xxxxxMatchNothingxxxxx -bench=.
.PHONY: golangci-lint golangci-lint-fix
golangci-lint-fix: ARGS=--fix
golangci-lint-fix: golangci-lint
golangci-lint: $(OTEL_GO_MOD_DIRS:%=golangci-lint/%)
golangci-lint/%: DIR=$*
golangci-lint/%: | $(GOLANGCI_LINT)
golangci-lint/%: $(GOLANGCI_LINT)
@echo 'golangci-lint $(if $(ARGS),$(ARGS) ,)$(DIR)' \
&& cd $(DIR) \
&& $(GOLANGCI_LINT) run --allow-serial-runners $(ARGS)
.PHONY: crosslink
crosslink: | $(CROSSLINK)
crosslink: $(CROSSLINK)
@echo "Updating intra-repository dependencies in all go modules" \
&& $(CROSSLINK) --root=$(shell pwd) --prune
.PHONY: go-mod-tidy
go-mod-tidy: $(ALL_GO_MOD_DIRS:%=go-mod-tidy/%)
go-mod-tidy/%: DIR=$*
go-mod-tidy/%: | crosslink
go-mod-tidy/%: crosslink
@echo "$(GO) mod tidy in $(DIR)" \
&& cd $(DIR) \
&& $(GO) mod tidy -compat=1.19
.PHONY: lint-modules
lint-modules: go-mod-tidy
&& $(GO) mod tidy -compat=1.21
.PHONY: lint
lint: misspell lint-modules golangci-lint
lint: misspell go-mod-tidy golangci-lint govulncheck
.PHONY: vanity-import-check
vanity-import-check: | $(PORTO)
@$(PORTO) --include-internal -l . || echo "(run: make vanity-import-fix)"
.PHONY: vanity-import-fix
vanity-import-fix: | $(PORTO)
@$(PORTO) --include-internal -w .
vanity-import-check: $(PORTO)
@$(PORTO) --include-internal -l . || ( echo "(run: make vanity-import-fix)"; exit 1 )
.PHONY: misspell
misspell: | $(MISSPELL)
misspell: $(MISSPELL)
@$(MISSPELL) -w $(ALL_DOCS)
.PHONY: govulncheck
govulncheck: $(OTEL_GO_MOD_DIRS:%=govulncheck/%)
govulncheck/%: DIR=$*
govulncheck/%: $(GOVULNCHECK)
@echo "govulncheck ./... in $(DIR)" \
&& cd $(DIR) \
&& $(GOVULNCHECK) ./...
.PHONY: codespell
codespell: | $(CODESPELL)
codespell: $(CODESPELL)
@$(DOCKERPY) $(CODESPELL)
.PHONY: toolchain-check
toolchain-check:
@toolchainRes=$$(for f in $(ALL_GO_MOD_DIRS); do \
awk '/^toolchain/ { found=1; next } END { if (found) print FILENAME }' $$f/go.mod; \
done); \
if [ -n "$${toolchainRes}" ]; then \
echo "toolchain checking failed:"; echo "$${toolchainRes}"; \
exit 1; \
fi
.PHONY: license-check
license-check:
@licRes=$$(for f in $$(find . -type f \( -iname '*.go' -o -iname '*.sh' \) ! -path '**/third_party/*' ! -path './.git/*' ) ; do \
awk '/Copyright The OpenTelemetry Authors|generated|GENERATED/ && NR<=3 { found=1; next } END { if (!found) print FILENAME }' $$f; \
awk '/Copyright The OpenTelemetry Authors|generated|GENERATED/ && NR<=4 { found=1; next } END { if (!found) print FILENAME }' $$f; \
done); \
if [ -n "$${licRes}" ]; then \
echo "license header checking failed:"; echo "$${licRes}"; \
exit 1; \
fi
DEPENDABOT_CONFIG = .github/dependabot.yml
.PHONY: dependabot-check
dependabot-check: | $(DBOTCONF)
@$(DBOTCONF) verify $(DEPENDABOT_CONFIG) || echo "(run: make dependabot-generate)"
.PHONY: dependabot-generate
dependabot-generate: | $(DBOTCONF)
@$(DBOTCONF) generate > $(DEPENDABOT_CONFIG)
.PHONY: check-clean-work-tree
check-clean-work-tree:
@if ! git diff --quiet; then \
@@ -246,24 +266,61 @@ check-clean-work-tree:
exit 1; \
fi
# The weaver docker image to use for semconv-generate.
WEAVER_IMAGE := $(shell awk '$$4=="weaver" {print $$2}' $(DEPENDENCIES_DOCKERFILE))
SEMCONVPKG ?= "semconv/"
.PHONY: semconv-generate
semconv-generate: | $(SEMCONVGEN) $(SEMCONVKIT)
[ "$(TAG)" ] || ( echo "TAG unset: missing opentelemetry specification tag"; exit 1 )
[ "$(OTEL_SPEC_REPO)" ] || ( echo "OTEL_SPEC_REPO unset: missing path to opentelemetry specification repo"; exit 1 )
$(SEMCONVGEN) -i "$(OTEL_SPEC_REPO)/semantic_conventions/." --only=span -p conventionType=trace -f trace.go -t "$(SEMCONVPKG)/template.j2" -s "$(TAG)"
$(SEMCONVGEN) -i "$(OTEL_SPEC_REPO)/semantic_conventions/." --only=attribute_group -p conventionType=trace -f attribute_group.go -t "$(SEMCONVPKG)/template.j2" -s "$(TAG)"
$(SEMCONVGEN) -i "$(OTEL_SPEC_REPO)/semantic_conventions/." --only=event -p conventionType=event -f event.go -t "$(SEMCONVPKG)/template.j2" -s "$(TAG)"
$(SEMCONVGEN) -i "$(OTEL_SPEC_REPO)/semantic_conventions/." --only=resource -p conventionType=resource -f resource.go -t "$(SEMCONVPKG)/template.j2" -s "$(TAG)"
$(SEMCONVKIT) -output "$(SEMCONVPKG)/$(TAG)" -tag "$(TAG)"
semconv-generate: $(SEMCONVKIT)
[ "$(TAG)" ] || ( echo "TAG unset: missing opentelemetry semantic-conventions tag"; exit 1 )
# Ensure the target directory for source code is available.
mkdir -p $(PWD)/$(SEMCONVPKG)/${TAG}
# Note: We mount a home directory for downloading/storing the semconv repository.
# Weaver will automatically clean the cache when finished, but the directories will remain.
mkdir -p ~/.weaver
docker run --rm \
-u $(DOCKER_USER) \
--env HOME=/tmp/weaver \
--mount 'type=bind,source=$(PWD)/semconv/templates,target=/home/weaver/templates,readonly' \
--mount 'type=bind,source=$(PWD)/semconv/${TAG},target=/home/weaver/target' \
--mount 'type=bind,source=$(HOME)/.weaver,target=/tmp/weaver/.weaver' \
$(WEAVER_IMAGE) registry generate \
--registry=https://github.com/open-telemetry/semantic-conventions/archive/refs/tags/$(TAG).zip[model] \
--templates=/home/weaver/templates \
--param tag=$(TAG) \
go \
/home/weaver/target
$(SEMCONVKIT) -semconv "$(SEMCONVPKG)" -tag "$(TAG)"
.PHONY: gorelease
gorelease: $(OTEL_GO_MOD_DIRS:%=gorelease/%)
gorelease/%: DIR=$*
gorelease/%:| $(GORELEASE)
@echo "gorelease in $(DIR):" \
&& cd $(DIR) \
&& $(GORELEASE) \
|| echo ""
.PHONY: verify-mods
verify-mods: $(MULTIMOD)
$(MULTIMOD) verify
.PHONY: prerelease
prerelease: | $(MULTIMOD)
prerelease: verify-mods
@[ "${MODSET}" ] || ( echo ">> env var MODSET is not set"; exit 1 )
$(MULTIMOD) verify && $(MULTIMOD) prerelease -m ${MODSET}
$(MULTIMOD) prerelease -m ${MODSET}
COMMIT ?= "HEAD"
.PHONY: add-tags
add-tags: | $(MULTIMOD)
add-tags: verify-mods
@[ "${MODSET}" ] || ( echo ">> env var MODSET is not set"; exit 1 )
$(MULTIMOD) verify && $(MULTIMOD) tag -m ${MODSET} -c ${COMMIT}
$(MULTIMOD) tag -m ${MODSET} -c ${COMMIT}
MARKDOWNIMAGE := $(shell awk '$$4=="markdown" {print $$2}' $(DEPENDENCIES_DOCKERFILE))
.PHONY: lint-markdown
lint-markdown:
docker run --rm -u $(DOCKER_USER) -v "$(CURDIR):$(WORKDIR)" $(MARKDOWNIMAGE) -c $(WORKDIR)/.markdownlint.yaml $(WORKDIR)/**/*.md
.PHONY: verify-readmes
verify-readmes: $(VERIFYREADMES)
$(VERIFYREADMES)