Major updates and improvements to BZZZ system
- Updated configuration and deployment files - Improved system architecture and components - Enhanced documentation and testing - Fixed various issues and added new features 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
anthonyrawlins
373
deployments/internal-docker/README-HCFS-Integration.md
Normal file
373
deployments/internal-docker/README-HCFS-Integration.md
Normal file
@@ -0,0 +1,373 @@
|
||||
# HCFS-Integrated Development Environment
|
||||
|
||||
This directory contains Docker configurations for creating HCFS-enabled development environments that provide AI agents with persistent, context-aware workspaces.
|
||||
|
||||
## 🎯 Overview
|
||||
|
||||
Instead of using temporary directories that are lost when containers stop, this system integrates with HCFS (Hierarchical Context File System) to provide:
|
||||
|
||||
- **Persistent Workspaces**: Agent work is stored in HCFS and survives container restarts
|
||||
- **Context Sharing**: Multiple agents can access and build upon each other's work
|
||||
- **Intelligent Artifact Collection**: Important files are automatically stored in HCFS
|
||||
- **Role-Based Access**: Agents can access context relevant to their specialization
|
||||
- **Feedback Learning**: The RL Context Curator learns from agent success/failure patterns
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Bzzz Agents │ │ HCFS-Enabled │ │ HCFS Core │
|
||||
│ │ │ Containers │ │ │
|
||||
│ • CLI Agents │◄──►│ │◄──►│ • Context API │
|
||||
│ • Ollama Models │ │ • Python Dev │ │ • RL Curator │
|
||||
│ • Reasoning │ │ • Node.js Dev │ │ • Storage │
|
||||
│ • Code Review │ │ • Go Dev │ │ • Search │
|
||||
└─────────────────┘ │ • Generic Base │ └─────────────────┘
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
## 🐳 Available Images
|
||||
|
||||
### Base Image: `bzzz-hcfs-base`
|
||||
- Ubuntu 22.04 with HCFS integration
|
||||
- Standard development tools (git, make, curl, etc.)
|
||||
- HCFS workspace management scripts
|
||||
- Agent user with proper permissions
|
||||
- FUSE support for HCFS mounting
|
||||
|
||||
### Language-Specific Images:
|
||||
|
||||
#### `bzzz-hcfs-python`
|
||||
- Python 3.10 with comprehensive ML/AI packages
|
||||
- Jupyter Lab/Notebook support
|
||||
- Popular frameworks: Flask, FastAPI, Django
|
||||
- Data science stack: NumPy, Pandas, scikit-learn
|
||||
- Deep learning: PyTorch, Transformers
|
||||
- **Ports**: 8888 (Jupyter), 8000, 5000, 8080
|
||||
|
||||
#### `bzzz-hcfs-nodejs`
|
||||
- Node.js 20 with modern JavaScript/TypeScript tools
|
||||
- Package managers: npm, yarn
|
||||
- Build tools: Webpack, Vite, Rollup
|
||||
- Testing: Jest, Mocha, Cypress
|
||||
- **Ports**: 3000, 8080, 8000, 9229 (debugger)
|
||||
|
||||
#### `bzzz-hcfs-go`
|
||||
- Go 1.21 with standard development tools
|
||||
- Popular frameworks: Gin, Echo, Fiber
|
||||
- Development tools: Delve debugger, Air live reload
|
||||
- **Ports**: 8080, 8000, 9000, 2345 (debugger)
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### 1. Build the Images
|
||||
```bash
|
||||
cd /home/tony/AI/projects/Bzzz/docker
|
||||
./build-hcfs-images.sh build
|
||||
```
|
||||
|
||||
### 2. Start the HCFS Ecosystem
|
||||
```bash
|
||||
docker-compose -f docker-compose.hcfs.yml up -d
|
||||
```
|
||||
|
||||
### 3. Access Development Environments
|
||||
|
||||
**Python Development:**
|
||||
```bash
|
||||
# Interactive shell
|
||||
docker exec -it agent-python-dev bash
|
||||
|
||||
# Jupyter Lab
|
||||
open http://localhost:8888
|
||||
```
|
||||
|
||||
**Node.js Development:**
|
||||
```bash
|
||||
# Interactive shell
|
||||
docker exec -it agent-nodejs-dev bash
|
||||
|
||||
# Start development server
|
||||
docker exec -it agent-nodejs-dev npm run dev
|
||||
```
|
||||
|
||||
**Go Development:**
|
||||
```bash
|
||||
# Interactive shell
|
||||
docker exec -it agent-go-dev bash
|
||||
|
||||
# Build and run
|
||||
docker exec -it agent-go-dev make build run
|
||||
```
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
**Required for HCFS Integration:**
|
||||
- `AGENT_ID`: Unique identifier for the agent
|
||||
- `TASK_ID`: Task identifier for workspace context
|
||||
- `HCFS_API_URL`: HCFS API endpoint (default: http://host.docker.internal:8000)
|
||||
- `HCFS_ENABLED`: Enable/disable HCFS integration (default: true)
|
||||
|
||||
**Optional:**
|
||||
- `GIT_USER_NAME`: Git configuration
|
||||
- `GIT_USER_EMAIL`: Git configuration
|
||||
- `SETUP_PYTHON_VENV`: Create Python virtual environment
|
||||
- `NODE_ENV`: Node.js environment mode
|
||||
|
||||
### HCFS Configuration
|
||||
|
||||
Each container includes `/etc/hcfs/hcfs-agent.yaml` with:
|
||||
- API endpoints and timeouts
|
||||
- Workspace settings
|
||||
- Artifact collection patterns
|
||||
- Security configurations
|
||||
- Logging preferences
|
||||
|
||||
## 💾 Workspace Management
|
||||
|
||||
### Automatic Features
|
||||
|
||||
1. **Workspace Initialization**: Creates HCFS context for agent workspace
|
||||
2. **Continuous Sync**: Background daemon syncs workspace state every 30 seconds
|
||||
3. **Artifact Collection**: Automatically stores important files:
|
||||
- Log files (*.log)
|
||||
- Documentation (*.md, README*)
|
||||
- Configuration (*.json, *.yaml)
|
||||
- Build outputs (build/*, output/*)
|
||||
- Results (results/*)
|
||||
|
||||
4. **Graceful Shutdown**: Collects final artifacts when container stops
|
||||
|
||||
### Manual Commands
|
||||
|
||||
```bash
|
||||
# Sync current workspace state
|
||||
/opt/hcfs/hcfs-workspace.sh sync
|
||||
|
||||
# Collect and store artifacts
|
||||
/opt/hcfs/hcfs-workspace.sh collect
|
||||
|
||||
# Finalize workspace (run on completion)
|
||||
/opt/hcfs/hcfs-workspace.sh finalize
|
||||
|
||||
# Check workspace status
|
||||
/opt/hcfs/hcfs-workspace.sh status
|
||||
```
|
||||
|
||||
## 🔄 Integration with Bzzz Agents
|
||||
|
||||
### Updated Sandbox Creation
|
||||
|
||||
The Bzzz sandbox system now supports HCFS workspaces:
|
||||
|
||||
```go
|
||||
// Create HCFS-enabled sandbox
|
||||
sandbox, err := CreateSandboxWithHCFS(ctx, taskImage, agentConfig, agentID, taskID)
|
||||
|
||||
// Check if using HCFS
|
||||
if sandbox.IsUsingHCFS() {
|
||||
workspace := sandbox.GetHCFSWorkspace()
|
||||
fmt.Printf("Using HCFS workspace: %s\n", workspace.HCFSPath)
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration in Bzzz
|
||||
|
||||
Add HCFS configuration to your Bzzz agent config:
|
||||
|
||||
```yaml
|
||||
hcfs:
|
||||
enabled: true
|
||||
api_url: "http://localhost:8000"
|
||||
mount_path: "/tmp/hcfs-workspaces"
|
||||
store_artifacts: true
|
||||
idle_cleanup_interval: "15m"
|
||||
max_idle_time: "1h"
|
||||
```
|
||||
|
||||
## 📊 Monitoring and Debugging
|
||||
|
||||
### Service Health Checks
|
||||
|
||||
```bash
|
||||
# Check HCFS API
|
||||
curl http://localhost:8000/health
|
||||
|
||||
# Check RL Tuner
|
||||
curl http://localhost:8001/health
|
||||
|
||||
# View container logs
|
||||
docker-compose -f docker-compose.hcfs.yml logs -f hcfs-api
|
||||
```
|
||||
|
||||
### Workspace Status
|
||||
|
||||
```bash
|
||||
# View workspace metadata
|
||||
cat /home/agent/work/.hcfs-workspace
|
||||
|
||||
# Check sync daemon status
|
||||
ps aux | grep hcfs-workspace
|
||||
|
||||
# View HCFS logs
|
||||
tail -f /var/log/hcfs/workspace.log
|
||||
```
|
||||
|
||||
## 🛠️ Development Workflows
|
||||
|
||||
### Python ML Development
|
||||
|
||||
```bash
|
||||
# Start Python environment
|
||||
docker exec -it agent-python-dev bash
|
||||
|
||||
# Create new project
|
||||
cd /home/agent/work
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Start Jupyter for data exploration
|
||||
jupyter lab --ip=0.0.0.0 --port=8888
|
||||
|
||||
# Artifacts automatically collected:
|
||||
# - *.ipynb notebooks
|
||||
# - model files in models/
|
||||
# - results in output/
|
||||
```
|
||||
|
||||
### Node.js Web Development
|
||||
|
||||
```bash
|
||||
# Start Node.js environment
|
||||
docker exec -it agent-nodejs-dev bash
|
||||
|
||||
# Initialize project
|
||||
cd /home/agent/work
|
||||
cp package.json.template package.json
|
||||
npm install
|
||||
|
||||
# Start development server
|
||||
npm run dev
|
||||
|
||||
# Artifacts automatically collected:
|
||||
# - package*.json
|
||||
# - build output in dist/
|
||||
# - logs in logs/
|
||||
```
|
||||
|
||||
### Go Microservices
|
||||
|
||||
```bash
|
||||
# Start Go environment
|
||||
docker exec -it agent-go-dev bash
|
||||
|
||||
# Initialize project
|
||||
cd /home/agent/work
|
||||
cp go.mod.template go.mod
|
||||
cp main.go.template main.go
|
||||
go mod tidy
|
||||
|
||||
# Build and run
|
||||
make build
|
||||
make run
|
||||
|
||||
# Artifacts automatically collected:
|
||||
# - go.mod, go.sum
|
||||
# - binary in bin/
|
||||
# - test results
|
||||
```
|
||||
|
||||
## 🔒 Security Considerations
|
||||
|
||||
### Container Security
|
||||
|
||||
- Agents run as non-root `agent` user
|
||||
- Limited sudo access only for FUSE mounts
|
||||
- Network restrictions block sensitive ports
|
||||
- Read-only access to system directories
|
||||
|
||||
### HCFS Security
|
||||
|
||||
- Context access controlled by agent roles
|
||||
- Workspace isolation between agents
|
||||
- Artifact encryption (optional)
|
||||
- Audit logging of all operations
|
||||
|
||||
## 🔄 Backup and Recovery
|
||||
|
||||
### Workspace Persistence
|
||||
|
||||
Agent workspaces are stored in named Docker volumes:
|
||||
- `python-workspace`: Python development files
|
||||
- `nodejs-workspace`: Node.js development files
|
||||
- `go-workspace`: Go development files
|
||||
|
||||
### HCFS Data
|
||||
|
||||
Core HCFS data is stored in:
|
||||
- `hcfs-data`: Main context database
|
||||
- `hcfs-rl-data`: RL Context Curator data
|
||||
|
||||
### Backup Script
|
||||
|
||||
```bash
|
||||
# Backup all workspace data
|
||||
docker run --rm -v python-workspace:/data -v /backup:/backup alpine \
|
||||
tar czf /backup/python-workspace-$(date +%Y%m%d).tar.gz -C /data .
|
||||
```
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**HCFS API Not Available:**
|
||||
```bash
|
||||
# Check if HCFS container is running
|
||||
docker ps | grep hcfs-api
|
||||
|
||||
# Check network connectivity
|
||||
docker exec agent-python-dev curl -f http://hcfs-api:8000/health
|
||||
```
|
||||
|
||||
**FUSE Mount Failures:**
|
||||
```bash
|
||||
# Check FUSE support
|
||||
docker exec agent-python-dev ls -la /dev/fuse
|
||||
|
||||
# Check mount permissions
|
||||
docker exec agent-python-dev mount | grep fuse
|
||||
```
|
||||
|
||||
**Workspace Sync Issues:**
|
||||
```bash
|
||||
# Restart sync daemon
|
||||
docker exec agent-python-dev pkill -f hcfs-workspace
|
||||
docker exec agent-python-dev /opt/hcfs/hcfs-workspace.sh daemon &
|
||||
|
||||
# Manual sync
|
||||
docker exec agent-python-dev /opt/hcfs/hcfs-workspace.sh sync
|
||||
```
|
||||
|
||||
### Log Locations
|
||||
|
||||
- HCFS API: `docker logs hcfs-api`
|
||||
- Agent containers: `docker logs agent-python-dev`
|
||||
- Workspace sync: `/var/log/hcfs/workspace.log` (inside container)
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- [HCFS Documentation](../HCFS/README.md)
|
||||
- [Bzzz Agent Configuration](../README.md)
|
||||
- [RL Context Curator Guide](../HCFS/integration_tests/README.md)
|
||||
- [Docker Compose Reference](https://docs.docker.com/compose/)
|
||||
|
||||
## 🎯 Next Steps
|
||||
|
||||
1. **Deploy to Production**: Use Docker Swarm or Kubernetes
|
||||
2. **Scale Horizontally**: Add more agent instances
|
||||
3. **Custom Images**: Create domain-specific development environments
|
||||
4. **Monitoring**: Add Prometheus/Grafana for metrics
|
||||
5. **CI/CD Integration**: Automate testing and deployment
|
||||
Reference in New Issue
Block a user