bzzz/deployments/internal-docker/README-HCFS-Integration.md

# HCFS-Integrated Development Environment

This directory contains Docker configurations for creating HCFS-enabled development environments that provide AI agents with persistent, context-aware workspaces.

## 🎯 Overview

Instead of using temporary directories that are lost when containers stop, this system integrates with HCFS (Hierarchical Context File System) to provide:

- **Persistent Workspaces**: Agent work is stored in HCFS and survives container restarts
- **Context Sharing**: Multiple agents can access and build upon each other's work
- **Intelligent Artifact Collection**: Important files are automatically stored in HCFS
- **Role-Based Access**: Agents can access context relevant to their specialization
- **Feedback Learning**: The RL Context Curator learns from agent success/failure patterns

## 🏗️ Architecture

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Bzzz Agents    │    │ HCFS-Enabled    │    │ HCFS Core       │
│                 │    │ Containers      │    │                 │
│ • CLI Agents    │◄──►│                 │◄──►│ • Context API   │
│ • Ollama Models │    │ • Python Dev    │    │ • RL Curator    │
│ • Reasoning     │    │ • Node.js Dev   │    │ • Storage       │
│ • Code Review   │    │ • Go Dev        │    │ • Search        │
└─────────────────┘    │ • Generic Base  │    └─────────────────┘
                       └─────────────────┘
```

## 🐳 Available Images

### Base Image: `bzzz-hcfs-base`
- Ubuntu 22.04 with HCFS integration
- Standard development tools (git, make, curl, etc.)
- HCFS workspace management scripts
- Agent user with proper permissions
- FUSE support for HCFS mounting

### Language-Specific Images:

#### `bzzz-hcfs-python`
- Python 3.10 with comprehensive ML/AI packages
- Jupyter Lab/Notebook support
- Popular frameworks: Flask, FastAPI, Django
- Data science stack: NumPy, Pandas, scikit-learn
- Deep learning: PyTorch, Transformers
- **Ports**: 8888 (Jupyter), 8000, 5000, 8080

#### `bzzz-hcfs-nodejs`
- Node.js 20 with modern JavaScript/TypeScript tools
- Package managers: npm, yarn
- Build tools: Webpack, Vite, Rollup
- Testing: Jest, Mocha, Cypress
- **Ports**: 3000, 8080, 8000, 9229 (debugger)

#### `bzzz-hcfs-go`
- Go 1.21 with standard development tools
- Popular frameworks: Gin, Echo, Fiber
- Development tools: Delve debugger, Air live reload
- **Ports**: 8080, 8000, 9000, 2345 (debugger)

## 🚀 Quick Start

### 1. Build the Images
```bash
cd /home/tony/AI/projects/Bzzz/docker
./build-hcfs-images.sh build
```

### 2. Start the HCFS Ecosystem
```bash
docker-compose -f docker-compose.hcfs.yml up -d
```

### 3. Access Development Environments

**Python Development:**
```bash
# Interactive shell
docker exec -it agent-python-dev bash

# Jupyter Lab
open http://localhost:8888
```

**Node.js Development:**
```bash
# Interactive shell  
docker exec -it agent-nodejs-dev bash

# Start development server
docker exec -it agent-nodejs-dev npm run dev
```

**Go Development:**
```bash
# Interactive shell
docker exec -it agent-go-dev bash

# Build and run
docker exec -it agent-go-dev make build run
```

## 🔧 Configuration

### Environment Variables

**Required for HCFS Integration:**
- `AGENT_ID`: Unique identifier for the agent
- `TASK_ID`: Task identifier for workspace context
- `HCFS_API_URL`: HCFS API endpoint (default: http://host.docker.internal:8000)
- `HCFS_ENABLED`: Enable/disable HCFS integration (default: true)

**Optional:**
- `GIT_USER_NAME`: Git configuration
- `GIT_USER_EMAIL`: Git configuration  
- `SETUP_PYTHON_VENV`: Create Python virtual environment
- `NODE_ENV`: Node.js environment mode

### HCFS Configuration

Each container includes `/etc/hcfs/hcfs-agent.yaml` with:
- API endpoints and timeouts
- Workspace settings
- Artifact collection patterns
- Security configurations
- Logging preferences

## 💾 Workspace Management

### Automatic Features

1. **Workspace Initialization**: Creates HCFS context for agent workspace
2. **Continuous Sync**: Background daemon syncs workspace state every 30 seconds
3. **Artifact Collection**: Automatically stores important files:
   - Log files (*.log)
   - Documentation (*.md, README*)
   - Configuration (*.json, *.yaml)
   - Build outputs (build/*, output/*)
   - Results (results/*)

4. **Graceful Shutdown**: Collects final artifacts when container stops

### Manual Commands

```bash
# Sync current workspace state
/opt/hcfs/hcfs-workspace.sh sync

# Collect and store artifacts
/opt/hcfs/hcfs-workspace.sh collect

# Finalize workspace (run on completion)
/opt/hcfs/hcfs-workspace.sh finalize

# Check workspace status
/opt/hcfs/hcfs-workspace.sh status
```

## 🔄 Integration with Bzzz Agents

### Updated Sandbox Creation

The Bzzz sandbox system now supports HCFS workspaces:

```go
// Create HCFS-enabled sandbox
sandbox, err := CreateSandboxWithHCFS(ctx, taskImage, agentConfig, agentID, taskID)

// Check if using HCFS
if sandbox.IsUsingHCFS() {
    workspace := sandbox.GetHCFSWorkspace()
    fmt.Printf("Using HCFS workspace: %s\n", workspace.HCFSPath)
}
```

### Configuration in Bzzz

Add HCFS configuration to your Bzzz agent config:

```yaml
hcfs:
  enabled: true
  api_url: "http://localhost:8000"
  mount_path: "/tmp/hcfs-workspaces"
  store_artifacts: true
  idle_cleanup_interval: "15m"
  max_idle_time: "1h"
```

## 📊 Monitoring and Debugging

### Service Health Checks

```bash
# Check HCFS API
curl http://localhost:8000/health

# Check RL Tuner
curl http://localhost:8001/health

# View container logs
docker-compose -f docker-compose.hcfs.yml logs -f hcfs-api
```

### Workspace Status

```bash
# View workspace metadata
cat /home/agent/work/.hcfs-workspace

# Check sync daemon status
ps aux | grep hcfs-workspace

# View HCFS logs
tail -f /var/log/hcfs/workspace.log
```

## 🛠️ Development Workflows

### Python ML Development

```bash
# Start Python environment
docker exec -it agent-python-dev bash

# Create new project
cd /home/agent/work
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Start Jupyter for data exploration
jupyter lab --ip=0.0.0.0 --port=8888

# Artifacts automatically collected:
# - *.ipynb notebooks
# - model files in models/
# - results in output/
```

### Node.js Web Development

```bash
# Start Node.js environment
docker exec -it agent-nodejs-dev bash

# Initialize project
cd /home/agent/work
cp package.json.template package.json
npm install

# Start development server
npm run dev

# Artifacts automatically collected:
# - package*.json
# - build output in dist/
# - logs in logs/
```

### Go Microservices

```bash
# Start Go environment
docker exec -it agent-go-dev bash

# Initialize project
cd /home/agent/work
cp go.mod.template go.mod
cp main.go.template main.go
go mod tidy

# Build and run
make build
make run

# Artifacts automatically collected:
# - go.mod, go.sum
# - binary in bin/
# - test results
```

## 🔒 Security Considerations

### Container Security

- Agents run as non-root `agent` user
- Limited sudo access only for FUSE mounts
- Network restrictions block sensitive ports
- Read-only access to system directories

### HCFS Security

- Context access controlled by agent roles
- Workspace isolation between agents
- Artifact encryption (optional)
- Audit logging of all operations

## 🔄 Backup and Recovery

### Workspace Persistence

Agent workspaces are stored in named Docker volumes:
- `python-workspace`: Python development files
- `nodejs-workspace`: Node.js development files  
- `go-workspace`: Go development files

### HCFS Data

Core HCFS data is stored in:
- `hcfs-data`: Main context database
- `hcfs-rl-data`: RL Context Curator data

### Backup Script

```bash
# Backup all workspace data
docker run --rm -v python-workspace:/data -v /backup:/backup alpine \
  tar czf /backup/python-workspace-$(date +%Y%m%d).tar.gz -C /data .
```

## 🐛 Troubleshooting

### Common Issues

**HCFS API Not Available:**
```bash
# Check if HCFS container is running
docker ps | grep hcfs-api

# Check network connectivity
docker exec agent-python-dev curl -f http://hcfs-api:8000/health
```

**FUSE Mount Failures:**
```bash
# Check FUSE support
docker exec agent-python-dev ls -la /dev/fuse

# Check mount permissions
docker exec agent-python-dev mount | grep fuse
```

**Workspace Sync Issues:**
```bash
# Restart sync daemon
docker exec agent-python-dev pkill -f hcfs-workspace
docker exec agent-python-dev /opt/hcfs/hcfs-workspace.sh daemon &

# Manual sync
docker exec agent-python-dev /opt/hcfs/hcfs-workspace.sh sync
```

### Log Locations

- HCFS API: `docker logs hcfs-api`
- Agent containers: `docker logs agent-python-dev`
- Workspace sync: `/var/log/hcfs/workspace.log` (inside container)

## 📚 Additional Resources

- [HCFS Documentation](../HCFS/README.md)
- [Bzzz Agent Configuration](../README.md)
- [RL Context Curator Guide](../HCFS/integration_tests/README.md)
- [Docker Compose Reference](https://docs.docker.com/compose/)

## 🎯 Next Steps

1. **Deploy to Production**: Use Docker Swarm or Kubernetes
2. **Scale Horizontally**: Add more agent instances
3. **Custom Images**: Create domain-specific development environments
4. **Monitoring**: Add Prometheus/Grafana for metrics
5. **CI/CD Integration**: Automate testing and deployment