Major WHOOSH system refactoring and feature enhancements
- Migrated from HIVE branding to WHOOSH across all components - Enhanced backend API with new services: AI models, BZZZ integration, templates, members - Added comprehensive testing suite with security, performance, and integration tests - Improved frontend with new components for project setup, AI models, and team management - Updated MCP server implementation with WHOOSH-specific tools and resources - Enhanced deployment configurations with production-ready Docker setups - Added comprehensive documentation and setup guides - Implemented age encryption service and UCXL integration 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
anthonyrawlins
799
planning/IMPLEMENTATION_PLAN.md
Normal file
799
planning/IMPLEMENTATION_PLAN.md
Normal file
@@ -0,0 +1,799 @@
|
||||
# 📋 CCLI Implementation Plan
|
||||
|
||||
**Project**: Gemini CLI Agent Integration
|
||||
**Version**: 1.0
|
||||
**Last Updated**: July 10, 2025
|
||||
|
||||
## 🎯 Implementation Strategy
|
||||
|
||||
### Core Principle: **Non-Disruptive Addition**
|
||||
- CLI agents are **additive** to existing Ollama infrastructure
|
||||
- Zero impact on current 7-agent Ollama cluster
|
||||
- Graceful degradation if CLI agents fail
|
||||
- Easy rollback mechanism
|
||||
|
||||
---
|
||||
|
||||
## 📊 Phase 1: Environment Testing & Validation (Week 1)
|
||||
|
||||
### 🎯 **Objective**: Comprehensive testing of CLI connectivity and environment setup
|
||||
|
||||
#### **1.1 Automated Connectivity Testing**
|
||||
```bash
|
||||
# File: scripts/test-connectivity.sh
|
||||
#!/bin/bash
|
||||
|
||||
# Test SSH connectivity to both machines
|
||||
test_ssh_connection() {
|
||||
local host=$1
|
||||
echo "Testing SSH connection to $host..."
|
||||
ssh -o ConnectTimeout=5 $host "echo 'SSH OK'" || return 1
|
||||
}
|
||||
|
||||
# Test Gemini CLI availability and functionality
|
||||
test_gemini_cli() {
|
||||
local host=$1
|
||||
local node_version=$2
|
||||
echo "Testing Gemini CLI on $host with Node $node_version..."
|
||||
|
||||
ssh $host "source ~/.nvm/nvm.sh && nvm use $node_version && echo 'Test prompt' | gemini --model gemini-2.5-pro | head -3"
|
||||
}
|
||||
|
||||
# Performance testing
|
||||
benchmark_response_time() {
|
||||
local host=$1
|
||||
local node_version=$2
|
||||
echo "Benchmarking response time on $host..."
|
||||
|
||||
time ssh $host "source ~/.nvm/nvm.sh && nvm use $node_version && echo 'What is 2+2?' | gemini --model gemini-2.5-pro"
|
||||
}
|
||||
```
|
||||
|
||||
#### **1.2 Environment Configuration Testing**
|
||||
- **WALNUT**: Node v22.14.0 environment verification
|
||||
- **IRONWOOD**: Node v22.17.0 environment verification
|
||||
- SSH key authentication setup and testing
|
||||
- Concurrent connection limit testing
|
||||
|
||||
#### **1.3 Error Condition Testing**
|
||||
- Network interruption scenarios
|
||||
- CLI timeout handling
|
||||
- Invalid model parameter testing
|
||||
- Rate limiting behavior analysis
|
||||
|
||||
#### **1.4 Deliverables**
|
||||
- [ ] Comprehensive connectivity test suite
|
||||
- [ ] Performance baseline measurements
|
||||
- [ ] Error handling scenarios documented
|
||||
- [ ] SSH configuration templates
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Phase 2: CLI Agent Adapter Implementation (Week 2)
|
||||
|
||||
### 🎯 **Objective**: Create robust CLI agent adapters with proper error handling
|
||||
|
||||
#### **2.1 Core Adapter Classes**
|
||||
|
||||
```python
|
||||
# File: src/agents/gemini_cli_agent.py
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, Dict, Any
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
@dataclass
|
||||
class GeminiCliConfig:
|
||||
"""Configuration for Gemini CLI agent"""
|
||||
host: str
|
||||
node_path: str
|
||||
gemini_path: str
|
||||
node_version: str
|
||||
model: str = "gemini-2.5-pro"
|
||||
timeout: int = 300 # 5 minutes
|
||||
max_concurrent: int = 2
|
||||
|
||||
class GeminiCliAgent:
|
||||
"""Adapter for Google Gemini CLI execution via SSH"""
|
||||
|
||||
def __init__(self, config: GeminiCliConfig, specialization: str):
|
||||
self.config = config
|
||||
self.specialization = specialization
|
||||
self.active_tasks = 0
|
||||
self.logger = logging.getLogger(f"gemini_cli.{config.host}")
|
||||
|
||||
async def execute_task(self, prompt: str, **kwargs) -> Dict[str, Any]:
|
||||
"""Execute a task using Gemini CLI"""
|
||||
if self.active_tasks >= self.config.max_concurrent:
|
||||
raise Exception("Agent at maximum concurrent tasks")
|
||||
|
||||
self.active_tasks += 1
|
||||
try:
|
||||
return await self._execute_remote_cli(prompt, **kwargs)
|
||||
finally:
|
||||
self.active_tasks -= 1
|
||||
|
||||
async def _execute_remote_cli(self, prompt: str, **kwargs) -> Dict[str, Any]:
|
||||
"""Execute CLI command via SSH with proper environment setup"""
|
||||
command = self._build_cli_command(prompt, **kwargs)
|
||||
|
||||
# Execute with timeout and proper error handling
|
||||
result = await self._ssh_execute(command)
|
||||
|
||||
return {
|
||||
"response": result.stdout,
|
||||
"execution_time": result.duration,
|
||||
"model": self.config.model,
|
||||
"agent_id": f"{self.config.host}-gemini",
|
||||
"status": "completed" if result.returncode == 0 else "failed"
|
||||
}
|
||||
```
|
||||
|
||||
#### **2.2 SSH Execution Engine**
|
||||
|
||||
```python
|
||||
# File: src/executors/ssh_executor.py
|
||||
import asyncio
|
||||
import asyncssh
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
@dataclass
|
||||
class SSHResult:
|
||||
stdout: str
|
||||
stderr: str
|
||||
returncode: int
|
||||
duration: float
|
||||
|
||||
class SSHExecutor:
|
||||
"""Manages SSH connections and command execution"""
|
||||
|
||||
def __init__(self, connection_pool_size: int = 5):
|
||||
self.connection_pool = {}
|
||||
self.pool_size = connection_pool_size
|
||||
|
||||
async def execute(self, host: str, command: str, timeout: int = 300) -> SSHResult:
|
||||
"""Execute command on remote host with connection pooling"""
|
||||
conn = await self._get_connection(host)
|
||||
|
||||
start_time = asyncio.get_event_loop().time()
|
||||
try:
|
||||
result = await asyncio.wait_for(
|
||||
conn.run(command, check=False),
|
||||
timeout=timeout
|
||||
)
|
||||
duration = asyncio.get_event_loop().time() - start_time
|
||||
|
||||
return SSHResult(
|
||||
stdout=result.stdout,
|
||||
stderr=result.stderr,
|
||||
returncode=result.exit_status,
|
||||
duration=duration
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
raise Exception(f"SSH command timeout after {timeout}s")
|
||||
```
|
||||
|
||||
#### **2.3 Agent Factory and Registry**
|
||||
|
||||
```python
|
||||
# File: src/agents/cli_agent_factory.py
|
||||
from typing import Dict, List
|
||||
from .gemini_cli_agent import GeminiCliAgent, GeminiCliConfig
|
||||
|
||||
class CliAgentFactory:
|
||||
"""Factory for creating and managing CLI agents"""
|
||||
|
||||
PREDEFINED_AGENTS = {
|
||||
"walnut-gemini": GeminiCliConfig(
|
||||
host="walnut",
|
||||
node_path="/home/tony/.nvm/versions/node/v22.14.0/bin/node",
|
||||
gemini_path="/home/tony/.nvm/versions/node/v22.14.0/bin/gemini",
|
||||
node_version="v22.14.0",
|
||||
model="gemini-2.5-pro"
|
||||
),
|
||||
"ironwood-gemini": GeminiCliConfig(
|
||||
host="ironwood",
|
||||
node_path="/home/tony/.nvm/versions/node/v22.17.0/bin/node",
|
||||
gemini_path="/home/tony/.nvm/versions/node/v22.17.0/bin/gemini",
|
||||
node_version="v22.17.0",
|
||||
model="gemini-2.5-pro"
|
||||
)
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def create_agent(cls, agent_id: str, specialization: str) -> GeminiCliAgent:
|
||||
"""Create a CLI agent by ID"""
|
||||
config = cls.PREDEFINED_AGENTS.get(agent_id)
|
||||
if not config:
|
||||
raise ValueError(f"Unknown CLI agent: {agent_id}")
|
||||
|
||||
return GeminiCliAgent(config, specialization)
|
||||
```
|
||||
|
||||
#### **2.4 Deliverables**
|
||||
- [ ] `GeminiCliAgent` core adapter class
|
||||
- [ ] `SSHExecutor` with connection pooling
|
||||
- [ ] `CliAgentFactory` for agent creation
|
||||
- [ ] Comprehensive unit tests for all components
|
||||
- [ ] Error handling and logging framework
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Phase 3: Backend Integration (Week 3)
|
||||
|
||||
### 🎯 **Objective**: Integrate CLI agents into existing WHOOSH backend
|
||||
|
||||
#### **3.1 Agent Type Extension**
|
||||
|
||||
```python
|
||||
# File: backend/app/core/whoosh_coordinator.py
|
||||
class AgentType(Enum):
|
||||
KERNEL_DEV = "kernel_dev"
|
||||
PYTORCH_DEV = "pytorch_dev"
|
||||
PROFILER = "profiler"
|
||||
DOCS_WRITER = "docs_writer"
|
||||
TESTER = "tester"
|
||||
CLI_GEMINI = "cli_gemini" # NEW: CLI-based Gemini agent
|
||||
GENERAL_AI = "general_ai" # NEW: General AI specialization
|
||||
REASONING = "reasoning" # NEW: Reasoning specialization
|
||||
```
|
||||
|
||||
#### **3.2 Enhanced Agent Model**
|
||||
|
||||
```python
|
||||
# File: backend/app/models/agent.py
|
||||
from sqlalchemy import Column, String, Integer, Enum as SQLEnum, JSON
|
||||
|
||||
class Agent(Base):
|
||||
__tablename__ = "agents"
|
||||
|
||||
id = Column(String, primary_key=True)
|
||||
endpoint = Column(String, nullable=False)
|
||||
model = Column(String, nullable=False)
|
||||
specialty = Column(String, nullable=False)
|
||||
max_concurrent = Column(Integer, default=2)
|
||||
current_tasks = Column(Integer, default=0)
|
||||
|
||||
# NEW: Agent type and CLI-specific configuration
|
||||
agent_type = Column(SQLEnum(AgentType), default=AgentType.OLLAMA)
|
||||
cli_config = Column(JSON, nullable=True) # Store CLI-specific config
|
||||
|
||||
def to_dict(self):
|
||||
return {
|
||||
"id": self.id,
|
||||
"endpoint": self.endpoint,
|
||||
"model": self.model,
|
||||
"specialty": self.specialty,
|
||||
"max_concurrent": self.max_concurrent,
|
||||
"current_tasks": self.current_tasks,
|
||||
"agent_type": self.agent_type.value,
|
||||
"cli_config": self.cli_config
|
||||
}
|
||||
```
|
||||
|
||||
#### **3.3 Enhanced Task Execution Router**
|
||||
|
||||
```python
|
||||
# File: backend/app/core/whoosh_coordinator.py
|
||||
class WHOOSHCoordinator:
|
||||
async def execute_task(self, task: Task, agent: Agent) -> Dict:
|
||||
"""Execute task with proper agent type routing"""
|
||||
|
||||
# Route to appropriate executor based on agent type
|
||||
if agent.agent_type == AgentType.CLI_GEMINI:
|
||||
return await self._execute_cli_task(task, agent)
|
||||
else:
|
||||
return await self._execute_ollama_task(task, agent)
|
||||
|
||||
async def _execute_cli_task(self, task: Task, agent: Agent) -> Dict:
|
||||
"""Execute task on CLI-based agent"""
|
||||
from ..agents.cli_agent_factory import CliAgentFactory
|
||||
|
||||
cli_agent = CliAgentFactory.create_agent(agent.id, agent.specialty)
|
||||
|
||||
# Build prompt from task context
|
||||
prompt = self._build_task_prompt(task)
|
||||
|
||||
try:
|
||||
result = await cli_agent.execute_task(prompt)
|
||||
task.status = TaskStatus.COMPLETED
|
||||
task.result = result
|
||||
return result
|
||||
except Exception as e:
|
||||
task.status = TaskStatus.FAILED
|
||||
task.result = {"error": str(e)}
|
||||
return {"error": str(e)}
|
||||
```
|
||||
|
||||
#### **3.4 Agent Registration API Updates**
|
||||
|
||||
```python
|
||||
# File: backend/app/api/agents.py
|
||||
@router.post("/agents/cli")
|
||||
async def register_cli_agent(agent_data: Dict[str, Any]):
|
||||
"""Register a CLI-based agent"""
|
||||
|
||||
# Validate CLI-specific fields
|
||||
required_fields = ["id", "agent_type", "cli_config", "specialty"]
|
||||
for field in required_fields:
|
||||
if field not in agent_data:
|
||||
raise HTTPException(400, f"Missing required field: {field}")
|
||||
|
||||
# Create agent with CLI configuration
|
||||
agent = Agent(
|
||||
id=agent_data["id"],
|
||||
endpoint=f"cli://{agent_data['cli_config']['host']}",
|
||||
model=agent_data.get("model", "gemini-2.5-pro"),
|
||||
specialty=agent_data["specialty"],
|
||||
agent_type=AgentType.CLI_GEMINI,
|
||||
cli_config=agent_data["cli_config"],
|
||||
max_concurrent=agent_data.get("max_concurrent", 2)
|
||||
)
|
||||
|
||||
# Test CLI agent connectivity before registration
|
||||
success = await test_cli_agent_connectivity(agent)
|
||||
if not success:
|
||||
raise HTTPException(400, "CLI agent connectivity test failed")
|
||||
|
||||
# Register agent
|
||||
db.add(agent)
|
||||
db.commit()
|
||||
|
||||
return {"status": "success", "agent_id": agent.id}
|
||||
```
|
||||
|
||||
#### **3.5 Deliverables**
|
||||
- [ ] Extended `AgentType` enum with CLI agent types
|
||||
- [ ] Enhanced `Agent` model with CLI configuration support
|
||||
- [ ] Updated task execution router for mixed agent types
|
||||
- [ ] CLI agent registration API endpoint
|
||||
- [ ] Database migration scripts
|
||||
- [ ] Integration tests for mixed agent execution
|
||||
|
||||
---
|
||||
|
||||
## 🔌 Phase 4: MCP Server Updates (Week 4)
|
||||
|
||||
### 🎯 **Objective**: Enable MCP server to work with mixed agent types
|
||||
|
||||
#### **4.1 Enhanced Agent Discovery**
|
||||
|
||||
```typescript
|
||||
// File: mcp-server/src/whoosh-tools.ts
|
||||
class WHOOSHTools {
|
||||
async discoverAgents(): Promise<AgentInfo[]> {
|
||||
const agents = await this.whooshClient.getAgents();
|
||||
|
||||
// Support both Ollama and CLI agents
|
||||
return agents.map(agent => ({
|
||||
id: agent.id,
|
||||
type: agent.agent_type || 'ollama',
|
||||
model: agent.model,
|
||||
specialty: agent.specialty,
|
||||
endpoint: agent.endpoint,
|
||||
available: agent.current_tasks < agent.max_concurrent
|
||||
}));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### **4.2 Multi-Type Task Execution**
|
||||
|
||||
```typescript
|
||||
// File: mcp-server/src/whoosh-tools.ts
|
||||
async executeTaskOnAgent(agentId: string, task: TaskRequest): Promise<TaskResult> {
|
||||
const agent = await this.getAgentById(agentId);
|
||||
|
||||
switch (agent.agent_type) {
|
||||
case 'ollama':
|
||||
return this.executeOllamaTask(agent, task);
|
||||
|
||||
case 'cli_gemini':
|
||||
return this.executeCliTask(agent, task);
|
||||
|
||||
default:
|
||||
throw new Error(`Unsupported agent type: ${agent.agent_type}`);
|
||||
}
|
||||
}
|
||||
|
||||
private async executeCliTask(agent: AgentInfo, task: TaskRequest): Promise<TaskResult> {
|
||||
// Execute task via CLI agent API
|
||||
const response = await this.whooshClient.executeCliTask(agent.id, task);
|
||||
|
||||
return {
|
||||
agent_id: agent.id,
|
||||
model: agent.model,
|
||||
response: response.response,
|
||||
execution_time: response.execution_time,
|
||||
status: response.status
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
#### **4.3 Mixed Agent Coordination Tools**
|
||||
|
||||
```typescript
|
||||
// File: mcp-server/src/whoosh-tools.ts
|
||||
async coordinateMultiAgentTask(requirements: string): Promise<CoordinationResult> {
|
||||
const agents = await this.discoverAgents();
|
||||
|
||||
// Intelligent agent selection based on task requirements and agent types
|
||||
const selectedAgents = this.selectOptimalAgents(requirements, agents);
|
||||
|
||||
// Execute tasks on mixed agent types (Ollama + CLI)
|
||||
const results = await Promise.all(
|
||||
selectedAgents.map(agent =>
|
||||
this.executeTaskOnAgent(agent.id, {
|
||||
type: this.determineTaskType(requirements, agent),
|
||||
prompt: this.buildAgentSpecificPrompt(requirements, agent),
|
||||
context: requirements
|
||||
})
|
||||
)
|
||||
);
|
||||
|
||||
return this.aggregateResults(results);
|
||||
}
|
||||
```
|
||||
|
||||
#### **4.4 Deliverables**
|
||||
- [ ] Enhanced agent discovery for mixed types
|
||||
- [ ] Multi-type task execution support
|
||||
- [ ] Intelligent agent selection algorithms
|
||||
- [ ] CLI agent health monitoring
|
||||
- [ ] Updated MCP tool documentation
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Phase 5: Frontend UI Updates (Week 5)
|
||||
|
||||
### 🎯 **Objective**: Extend UI to support CLI agents with proper visualization
|
||||
|
||||
#### **5.1 Agent Management UI Extensions**
|
||||
|
||||
```typescript
|
||||
// File: frontend/src/components/agents/AgentCard.tsx
|
||||
interface AgentCardProps {
|
||||
agent: Agent;
|
||||
}
|
||||
|
||||
const AgentCard: React.FC<AgentCardProps> = ({ agent }) => {
|
||||
const getAgentTypeIcon = (type: string) => {
|
||||
switch (type) {
|
||||
case 'ollama':
|
||||
return <Server className="h-4 w-4" />;
|
||||
case 'cli_gemini':
|
||||
return <Terminal className="h-4 w-4" />;
|
||||
default:
|
||||
return <HelpCircle className="h-4 w-4" />;
|
||||
}
|
||||
};
|
||||
|
||||
const getAgentTypeBadge = (type: string) => {
|
||||
return type === 'cli_gemini' ?
|
||||
<Badge variant="secondary">CLI</Badge> :
|
||||
<Badge variant="default">API</Badge>;
|
||||
};
|
||||
|
||||
return (
|
||||
<Card>
|
||||
<CardContent>
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center space-x-2">
|
||||
{getAgentTypeIcon(agent.agent_type)}
|
||||
<h3>{agent.id}</h3>
|
||||
{getAgentTypeBadge(agent.agent_type)}
|
||||
</div>
|
||||
<AgentStatusIndicator agent={agent} />
|
||||
</div>
|
||||
|
||||
{agent.agent_type === 'cli_gemini' && (
|
||||
<CliAgentDetails config={agent.cli_config} />
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
);
|
||||
};
|
||||
```
|
||||
|
||||
#### **5.2 CLI Agent Registration Form**
|
||||
|
||||
```typescript
|
||||
// File: frontend/src/components/agents/CliAgentForm.tsx
|
||||
const CliAgentForm: React.FC = () => {
|
||||
const [formData, setFormData] = useState({
|
||||
id: '',
|
||||
host: '',
|
||||
node_version: '',
|
||||
model: 'gemini-2.5-pro',
|
||||
specialty: 'general_ai',
|
||||
max_concurrent: 2
|
||||
});
|
||||
|
||||
const handleSubmit = async (e: React.FormEvent) => {
|
||||
e.preventDefault();
|
||||
|
||||
const cliConfig = {
|
||||
host: formData.host,
|
||||
node_path: `/home/tony/.nvm/versions/node/${formData.node_version}/bin/node`,
|
||||
gemini_path: `/home/tony/.nvm/versions/node/${formData.node_version}/bin/gemini`,
|
||||
node_version: formData.node_version
|
||||
};
|
||||
|
||||
await registerCliAgent({
|
||||
...formData,
|
||||
agent_type: 'cli_gemini',
|
||||
cli_config: cliConfig
|
||||
});
|
||||
};
|
||||
|
||||
return (
|
||||
<form onSubmit={handleSubmit}>
|
||||
{/* Form fields for CLI agent configuration */}
|
||||
</form>
|
||||
);
|
||||
};
|
||||
```
|
||||
|
||||
#### **5.3 Mixed Agent Dashboard**
|
||||
|
||||
```typescript
|
||||
// File: frontend/src/pages/AgentsDashboard.tsx
|
||||
const AgentsDashboard: React.FC = () => {
|
||||
const [agents, setAgents] = useState<Agent[]>([]);
|
||||
|
||||
const groupedAgents = useMemo(() => {
|
||||
return agents.reduce((groups, agent) => {
|
||||
const type = agent.agent_type || 'ollama';
|
||||
if (!groups[type]) groups[type] = [];
|
||||
groups[type].push(agent);
|
||||
return groups;
|
||||
}, {} as Record<string, Agent[]>);
|
||||
}, [agents]);
|
||||
|
||||
return (
|
||||
<div>
|
||||
<h1>Agent Dashboard</h1>
|
||||
|
||||
{Object.entries(groupedAgents).map(([type, typeAgents]) => (
|
||||
<section key={type}>
|
||||
<h2>{type.toUpperCase()} Agents ({typeAgents.length})</h2>
|
||||
<div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-4">
|
||||
{typeAgents.map(agent => (
|
||||
<AgentCard key={agent.id} agent={agent} />
|
||||
))}
|
||||
</div>
|
||||
</section>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
};
|
||||
```
|
||||
|
||||
#### **5.4 Deliverables**
|
||||
- [ ] CLI agent visualization components
|
||||
- [ ] Mixed agent dashboard with type grouping
|
||||
- [ ] CLI agent registration and management forms
|
||||
- [ ] Enhanced monitoring displays for CLI agents
|
||||
- [ ] Responsive design for CLI-specific information
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Phase 6: Production Testing & Deployment (Week 6)
|
||||
|
||||
### 🎯 **Objective**: Comprehensive testing and safe production deployment
|
||||
|
||||
#### **6.1 Performance Testing**
|
||||
|
||||
```bash
|
||||
# File: scripts/benchmark-cli-agents.sh
|
||||
#!/bin/bash
|
||||
|
||||
echo "Benchmarking CLI vs Ollama Agent Performance"
|
||||
|
||||
# Test concurrent execution limits
|
||||
test_concurrent_limit() {
|
||||
local agent_type=$1
|
||||
local max_concurrent=$2
|
||||
|
||||
echo "Testing $max_concurrent concurrent tasks on $agent_type agents..."
|
||||
|
||||
for i in $(seq 1 $max_concurrent); do
|
||||
{
|
||||
curl -X POST http://localhost:8000/api/tasks \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"agent_type\": \"$agent_type\", \"prompt\": \"Test task $i\"}" &
|
||||
}
|
||||
done
|
||||
|
||||
wait
|
||||
echo "Concurrent test completed for $agent_type"
|
||||
}
|
||||
|
||||
# Response time comparison
|
||||
compare_response_times() {
|
||||
echo "Comparing response times..."
|
||||
|
||||
# Ollama agent baseline
|
||||
ollama_time=$(time_api_call "ollama" "What is the capital of France?")
|
||||
|
||||
# CLI agent comparison
|
||||
cli_time=$(time_api_call "cli_gemini" "What is the capital of France?")
|
||||
|
||||
echo "Ollama response time: ${ollama_time}s"
|
||||
echo "CLI response time: ${cli_time}s"
|
||||
}
|
||||
```
|
||||
|
||||
#### **6.2 Load Testing Suite**
|
||||
|
||||
```python
|
||||
# File: scripts/load_test_cli_agents.py
|
||||
import asyncio
|
||||
import aiohttp
|
||||
import time
|
||||
from typing import List, Dict
|
||||
|
||||
class CliAgentLoadTester:
|
||||
def __init__(self, base_url: str = "http://localhost:8000"):
|
||||
self.base_url = base_url
|
||||
|
||||
async def execute_concurrent_tasks(self, agent_id: str, num_tasks: int) -> List[Dict]:
|
||||
"""Execute multiple concurrent tasks on a CLI agent"""
|
||||
async with aiohttp.ClientSession() as session:
|
||||
tasks = []
|
||||
|
||||
for i in range(num_tasks):
|
||||
task = self.execute_single_task(session, agent_id, f"Task {i}")
|
||||
tasks.append(task)
|
||||
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
return results
|
||||
|
||||
async def stress_test(self, duration_minutes: int = 10):
|
||||
"""Run stress test for specified duration"""
|
||||
end_time = time.time() + (duration_minutes * 60)
|
||||
task_count = 0
|
||||
|
||||
while time.time() < end_time:
|
||||
# Alternate between CLI and Ollama agents
|
||||
agent_id = "walnut-gemini" if task_count % 2 == 0 else "walnut"
|
||||
|
||||
try:
|
||||
await self.execute_single_task_direct(agent_id, f"Stress test task {task_count}")
|
||||
task_count += 1
|
||||
except Exception as e:
|
||||
print(f"Task {task_count} failed: {e}")
|
||||
|
||||
print(f"Stress test completed: {task_count} tasks in {duration_minutes} minutes")
|
||||
```
|
||||
|
||||
#### **6.3 Production Deployment Strategy**
|
||||
|
||||
```yaml
|
||||
# File: config/production-deployment.yaml
|
||||
cli_agents:
|
||||
deployment_strategy: "blue_green"
|
||||
|
||||
agents:
|
||||
walnut-gemini:
|
||||
enabled: false # Start disabled
|
||||
priority: 1 # Lower priority initially
|
||||
max_concurrent: 1 # Conservative limit
|
||||
|
||||
ironwood-gemini:
|
||||
enabled: false
|
||||
priority: 1
|
||||
max_concurrent: 1
|
||||
|
||||
gradual_rollout:
|
||||
phase_1:
|
||||
duration_hours: 24
|
||||
enabled_agents: ["walnut-gemini"]
|
||||
traffic_percentage: 10
|
||||
|
||||
phase_2:
|
||||
duration_hours: 48
|
||||
enabled_agents: ["walnut-gemini", "ironwood-gemini"]
|
||||
traffic_percentage: 25
|
||||
|
||||
phase_3:
|
||||
duration_hours: 72
|
||||
enabled_agents: ["walnut-gemini", "ironwood-gemini"]
|
||||
traffic_percentage: 50
|
||||
```
|
||||
|
||||
#### **6.4 Monitoring and Alerting Setup**
|
||||
|
||||
```yaml
|
||||
# File: monitoring/cli-agent-alerts.yaml
|
||||
alerts:
|
||||
- name: "CLI Agent Response Time High"
|
||||
condition: "cli_agent_response_time > 30s"
|
||||
severity: "warning"
|
||||
|
||||
- name: "CLI Agent Failure Rate High"
|
||||
condition: "cli_agent_failure_rate > 10%"
|
||||
severity: "critical"
|
||||
|
||||
- name: "SSH Connection Pool Exhausted"
|
||||
condition: "ssh_connection_pool_usage > 90%"
|
||||
severity: "warning"
|
||||
|
||||
dashboards:
|
||||
- name: "CLI Agent Performance"
|
||||
panels:
|
||||
- response_time_comparison
|
||||
- success_rate_by_agent_type
|
||||
- concurrent_task_execution
|
||||
- ssh_connection_metrics
|
||||
```
|
||||
|
||||
#### **6.5 Deliverables**
|
||||
- [ ] Comprehensive load testing suite
|
||||
- [ ] Performance comparison reports
|
||||
- [ ] Production deployment scripts with gradual rollout
|
||||
- [ ] Monitoring dashboards for CLI agents
|
||||
- [ ] Alerting configuration for CLI agent issues
|
||||
- [ ] Rollback procedures and documentation
|
||||
|
||||
---
|
||||
|
||||
## 📊 Success Metrics
|
||||
|
||||
### **Technical Metrics**
|
||||
- **Response Time**: CLI agents average response time ≤ 150% of Ollama agents
|
||||
- **Success Rate**: CLI agent task success rate ≥ 95%
|
||||
- **Concurrent Execution**: Support ≥ 4 concurrent CLI tasks across both machines
|
||||
- **Availability**: CLI agent uptime ≥ 99%
|
||||
|
||||
### **Operational Metrics**
|
||||
- **Zero Downtime**: No impact on existing Ollama agent functionality
|
||||
- **Easy Rollback**: Ability to disable CLI agents within 5 minutes
|
||||
- **Monitoring Coverage**: 100% of CLI agent operations monitored and alerted
|
||||
|
||||
### **Business Metrics**
|
||||
- **Task Diversity**: 20% increase in supported task types
|
||||
- **Model Options**: Access to Google's Gemini 2.5 Pro capabilities
|
||||
- **Future Readiness**: Framework ready for additional CLI-based AI tools
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Risk Mitigation Plan
|
||||
|
||||
### **High Risk Items**
|
||||
1. **SSH Connection Stability**: Implement connection pooling and automatic reconnection
|
||||
2. **CLI Tool Updates**: Version pinning and automated testing of CLI tool updates
|
||||
3. **Rate Limiting**: Implement intelligent backoff and quota management
|
||||
4. **Security**: Secure key management and network isolation
|
||||
|
||||
### **Rollback Strategy**
|
||||
1. **Immediate**: Disable CLI agent registration endpoint
|
||||
2. **Short-term**: Mark all CLI agents as unavailable in database
|
||||
3. **Long-term**: Remove CLI agent code paths if needed
|
||||
|
||||
### **Testing Strategy**
|
||||
- **Unit Tests**: 90%+ coverage for CLI agent components
|
||||
- **Integration Tests**: End-to-end CLI agent execution testing
|
||||
- **Load Tests**: Sustained operation under production-like load
|
||||
- **Chaos Testing**: Network interruption and CLI tool failure scenarios
|
||||
|
||||
---
|
||||
|
||||
## 📅 Timeline Summary
|
||||
|
||||
| Phase | Duration | Key Deliverables |
|
||||
|-------|----------|------------------|
|
||||
| **Phase 1** | Week 1 | Environment testing, connectivity validation |
|
||||
| **Phase 2** | Week 2 | CLI agent adapters, SSH execution engine |
|
||||
| **Phase 3** | Week 3 | Backend integration, API updates |
|
||||
| **Phase 4** | Week 4 | MCP server updates, mixed agent support |
|
||||
| **Phase 5** | Week 5 | Frontend UI extensions, CLI agent management |
|
||||
| **Phase 6** | Week 6 | Production testing, deployment, monitoring |
|
||||
|
||||
**Total Duration**: 6 weeks
|
||||
**Go-Live Target**: August 21, 2025
|
||||
|
||||
---
|
||||
|
||||
This implementation plan provides a comprehensive roadmap for safely integrating Gemini CLI agents into the WHOOSH platform while maintaining the stability and performance of the existing system.
|
||||
Reference in New Issue
Block a user