MonkeyCode 测试策略与质量保障：AI 编程助手的全面质量工程实践指南
引言
"在 AI 时代，'测试'的定义正在被重新书写。"
传统的软件测试关注"代码是否按预期执行"，而 AI 编程助手的测试还需要回答一个更根本的问题："AI 的输出是否真正帮助了开发者？" 这意味着 MonkeyCode 的质量保障体系必须同时覆盖传统软件工程质量维度和 AI 特有的质量维度。
作为完全开源的 AI 编程助手（Apache License 2.0），MonkeyCode 的测试策略不仅服务于内部质量保证，更是一份公开的质量承诺——让每一位用户、贡献者和企业客户都能了解我们如何确保产品的可靠性、安全性和有效性。
本文将系统性地分享 MonkeyCode 从单元测试到 AI 效果评估的完整测试体系，包括测试金字塔设计、AI 输出质量评估方法、自动化测试基础设施，以及开源社区如何参与质量共建。
🎯 核心信息
- GitHub 仓库: https://github.com/monkeycode-ai/monkeycode
- 开源协议: Apache License 2.0
- 欢迎提交 Issue: Bug 报告请标记
bug标签- CI 状态:
一、AI 编程助手的质量挑战
1.1 传统测试 vs AI 测试的本质差异
┌─────────────────────────────────────────────────────────────┐
│ 传统软件 vs AI 系统的质量维度对比 │
│ │
│ ══════════════════════════════════════════════════════ │
│ │
│ 传统软件测试 AI 系统测试 │
│ ────────── ───────── │
│ ✅ 确定性输出 ⚠️ 概率性输出 │
│ (相同输入 → 相同输出) (相同输入 → 可能不同) │
│ │
│ ✅ 明确的正确答案 ⚠️ "好"是主观的 │
│ (pass/fail) (质量评分而非二元判断) │
│ │
│ ✅ 完整的覆盖率可度量 ⚠️ 输出空间无限大 │
│ (行/分支/路径覆盖) (无法穷举所有可能输出) │
│ │
│ ✅ 回归测试稳定 ⚠️ 模型更新导致行为变化 │
│ (测试用例长期有效) (需要持续重新评估) │
│ │
│ ✅ 性能基准明确 ⚠️ 质量与成本的权衡 │
│ (响应时间/吞吐量) (更好的质量 = 更多计算) │
│ │
│ ══════════════════════════════════════════════════════ │
│ │
│ 💡 关键洞察: AI 测试不是要替代传统测试， │
│ 而是在其基础上增加新的评估维度 │
│ │
└─────────────────────────────────────────────────────────────┘
1.2 MonkeyCode 质量矩阵
| 质量维度 | 子维度 | 度量指标 | 目标值 | 测试方法 |
|---|---|---|---|---|
| 功能性 | 补全准确率 | Accept@1, Accept@5 | >65%, >85% | 基准测试集 |
| 代码正确性 | 通过编译比例 | >95% | 自动编译验证 | |
| 功能完整性 | 需求覆盖率 | 100% | E2E 测试 | |
| 可靠性 | 崩溃率 | CRASH/1000次调用 | <0.1% | 错误监控 |
| 超时率 | Timeout/1000次调用 | <1% | 性能测试 | |
| 数据丢失率 | 数据损坏事件 | 0 | 数据完整性检查 | |
| 性能 | 首次补全延迟 | P50/P99 | <200ms/<800ms | 性能基准 |
| 启动时间 | 冷启动 | <3s | 启动性能测试 | |
| 内存占用 | RSS | <300MB | 内存分析 | |
| 安全性 | 注入攻击防护 | SQLi/XSS 阻断率 | 100% | 安全扫描 |
| 数据隐私 | 本地处理比例 | >99% | 审计日志 | |
| 权限控制 | 越权访问事件 | 0 | 权限测试 | |
| 可用性 | NPS 分数 | 用户满意度 | >60 | 用户调研 |
| 易上手时间 | Time-to-first-value | <5分钟 | 可用性测试 | |
| 错误恢复能力 | 自愈成功率 | >90% | 混沌测试 |
二、MonkeyCode 测试金字塔
2.1 四层测试架构
┌─────────────────────────────────────────────────────────────────────────┐
│ MonkeyCode 测试金字塔 │
│ │
│ ▲ │
│ /|\ │
│ / | \ │
│ / | \ │
│ / | \ │
│ / AI-E2E \ │
│ / Tests \ ← 少量但高价值 │
│ / (端到端) \ ~50 个场景 │
│ / \ 执行时间: ~30min │
│ /──────────────────\ │
│ / Integration \ │
│ / Tests \ ← 中等数量 │
│ / (集成测试) \ ~500 个用例 │
│ / \ 执行时间: ~10min │
│ /──────────────────────────────\ │
│ / Unit Tests \ │
│ / (单元测试) \ ← 大量快速 │
│ / \ ~5000+ 个用例 │
│ / \ 执行时间: ~3min │
│ /────────────────────────────────────────\ │
│ / Static Analysis \ │
│ / (静态分析) \ ← 最多最快 │
│ / \ Lint + Type Check │
│ └────────────────────────────────────────────┘ 执行时间: ~30s │
│ │
│ ══════════════════════════════════════════════════════════════════ │
│ │
│ 数量趋势: 静态分析 >>> 单元测试 >> 集成测试 > AI-E2E 测试 │
│ 执行速度: 静态分析 < 单元测试 < 集成测试 << AI-E2E 测试 │
│ 反馈速度: 静态分析(秒级) → 单元(分钟) → 集成(分钟) → E2E(小时) │
│ │
╚═══════════════════════════════════════════════════════════════════════╝
2.2 各层详细说明
第一层：静态分析 (Static Analysis)
# monkeycode/.eslintrc.yaml (部分配置)
# MonkeyCode 静态分析规则集
rules:
# === 类型安全 ===
"@typescript-eslint/no-explicit-any": "error"
"@typescript-eslint/explicit-function-return-type": "warn"
"@typescript-eslint/no-unused-vars": ["error", { "argsIgnorePattern": "^_" }]
# === 代码质量 ===
"complexity": ["warn", { "max": 15 }]
"max-lines-per-function": ["warn", { "max": 80, "skipBlankLines": true, "skipComments": true }]
"max-depth": ["error", { "max": 4 }]
"max-nested-callbacks": ["error", { "max": 3 }]
# === 安全相关 ===
"no-eval": "error"
"no-implied-eval": "error"
"no-new-func": "error"
"no-script-url": "error"
# === AI 相关特殊规则 ===
"monkeyCode/no-hardcoded-api-key": "error" # 自定义：禁止硬编码 API Key
"monkeyCode/no-sync-ai-call-in-main-thread": "error" # 自定义：主线程禁止同步 AI 调用
"monkeyCode/require-error-handling-for-ai": "error" # 自定义：AI 调用必须有错误处理
# === 测试相关 ===
"jest/no-disabled-tests": "warn"
"jest/no-focused-tests": "error"
"jest/valid-expect": "error"
// ===== monkeycode/src/core/__tests__/unit/completion-engine.test.ts =====
/**
* MonkeyCode 补全引擎 — 单元测试示例
*
* 展示如何对 AI 编程助手的核心组件进行单元测试，
* 包括 mock AI 服务、状态管理、缓存逻辑等。
*/
import { CompletionEngine } from '../completion-engine';
import { MockAIService } from './mocks/mock-ai-service';
import { InMemoryCache } from './mocks/in-memory-cache';
describe('CompletionEngine', () => {
let engine: CompletionEngine;
let mockAI: MockAIService;
let cache: InMemoryCache;
beforeEach(() => {
mockAI = new MockAIService();
cache = new InMemoryCache();
engine = new CompletionEngine({
aiService: mockAI,
cache,
maxCacheSize: 1000,
cacheTTL: 3600000, // 1 hour
});
});
afterEach(() => {
jest.clearAllMocks();
engine.dispose();
});
// ========== 基本功能测试 ==========
describe('basic completion', () => {
it('should return completion for valid request', async () => {
const request = {
filePath: '/test.ts',
language: 'typescript',
position: { line: 0, column: 6 },
content: 'function add(a, b) {\n return a + ',
};
mockAI.setMockResponse({
choices: [{ text: 'b;\n}' }],
model: 'test-model',
usage: { promptTokens: 10, completionTokens: 5 },
});
const result = await engine.complete(request);
expect(result).toBeDefined();
expect(result.completions).toHaveLength(1);
expect(result.completions[0].text).toBe('b;\n}');
expect(result.model).toBe('test-model');
});
it('should handle empty content gracefully', async () => {
const request = {
filePath: '/empty.js',
language: 'javascript',
position: { line: 0, column: 0 },
content: '',
};
const result = await engine.complete(request);
// 空文件应该返回空结果而不是报错
expect(result).toBeDefined();
expect(result.completions).toEqual([]);
});
});
// ========== 缓存行为测试 ==========
describe('caching behavior', () => {
it('should return cached result for identical request', async () => {
const request = {
filePath: '/cached.ts',
language: 'typescript',
position: { line: 0, column: 10 },
content: 'const hello = "hello"',
};
mockAI.setMockResponse({
choices: [{ text: 'world";' }],
model: 'test',
usage: { promptTokens: 5, completionTokens: 2 },
});
// 第一次调用
const result1 = await engine.complete(request);
expect(mockAI.callCount).toBe(1);
// 第二次调用应该命中缓存
const result2 = await engine.complete(request);
expect(mockAI.callCount).toBe(1); // 不应该再次调用 AI
expect(result2.completions[0].text).toBe(result1.completions[0].text);
});
it('should invalidate cache when content changes', async () => {
const baseRequest = {
filePath: '/cache-test.ts',
language: 'typescript',
position: { line: 0, column: 15 },
content: 'let x = "original"',
};
mockAI.setMockResponse({
choices: [{ text: '"' }],
model: 'test',
usage: { promptTokens: 5, completionTokens: 1 },
});
await engine.complete(baseRequest);
// 修改内容后应该产生新的请求
const modifiedRequest = {
...baseRequest,
content: 'let x = "modified"',
};
mockAI.setMockResponse({
choices: [{ text = '";' }],
model: 'test',
usage: { promptTokens: 5, completionTokens: 1 },
});
await engine.complete(modifiedRequest);
expect(mockAI.callCount).toBe(2); // 应该有两次 AI 调用
});
});
// ========== 错误处理测试 ==========
describe('error handling', () => {
it('should handle AI service timeout gracefully', async () => {
const request = {
filePath: '/timeout.ts',
language: 'typescript',
position: { line: 0, column: 0 },
content: 'some code',
};
mockAI.setMockError(new Error('Request timeout'));
const result = await engine.complete(request);
expect(result.error).toBeDefined();
expect(result.error.code).toBe('TIMEOUT');
expect(result.completions).toEqual([]);
// 不应该抛出异常
});
it('should handle rate limiting with retry', async () => {
const request = {
filePath: '/rate-limit.ts',
language: 'typescript',
position: { line: 0, column: 0 },
content: 'code here',
};
// 前两次返回 rate limit 错误
mockAI.setMockSequence([
new Error('Rate limit exceeded'),
new Error('Rate limit exceeded'),
{
choices: [{ text: 'result' }],
model: 'test',
usage: { promptTokens: 5, completionTokens: 1 },
},
]);
const result = await engine.complete(request);
expect(result.completions).toHaveLength(1);
expect(mockAI.callCount).toBe(3); // 2 次失败 + 1 次成功
});
it('should handle malformed AI response', async () => {
const request = {
filePath: '/malformed.ts',
language: 'typescript',
position: { line: 0, column: 0 },
content: 'code',
};
// 返回格式错误的响应
mockAI.setMalformedResponse({ invalid: 'data' });
const result = await engine.complete(request);
expect(result.error).toBeDefined();
expect(result.error.code).toBe('INVALID_RESPONSE');
});
});
// ========== 并发安全测试 ==========
describe('concurrency safety', () => {
it('should handle multiple concurrent requests', async () => {
const requests = Array.from({ length: 20 }, (_, i) => ({
filePath: `/concurrent-${i}.ts`,
language: 'typescript' as const,
position: { line: 0, column: i },
content: `const v${i} = `,
}));
mockAI.setMockResponse({
choices: [{ text: `${Math.random()};` }],
model: 'test',
usage: { promptTokens: 5, completionTokens: 1 },
});
// 并发发送 20 个请求
const results = await Promise.all(
requests.map(r => engine.complete(r))
);
// 所有请求都应该成功完成
expect(results.every(r => r.completions.length > 0)).toBe(true);
// AI 服务应该被调用了 20 次
expect(mockAI.callCount).toBe(20);
});
it('should deduplicate identical concurrent requests', async () => {
const sameRequest = {
filePath: '/dedup.ts',
language: 'typescript' as const,
position: { line: 0, column: 5 },
content: 'const x = ',
};
mockAI.setMockResponse({
choices: [{ text: '1;' }],
model: 'test',
usage: { promptTokens: 5, completionTokens: 1 },
});
// 同时发送 5 个相同的请求
const promises = Array.from({ length: 5 }, () =>
engine.complete(sameRequest)
);
const results = await Promise.all(promises);
// 由于去重机制，AI 只应该被调用一次
expect(mockAI.callCount).toBeLessThanOrEqual(2);
// 所有结果都应该一致
const texts = results.map(r => r.completions[0]?.text);
expect(new Set(texts).size).toBe(1);
});
});
});
第二层：集成测试 (Integration Tests)
# ===== monkeycode/tests/integration/test_editor_integration.py =====
"""
MonkeyCode 编辑器集成测试
测试 MonkeyCode 与 VSCode/JetBrains 编辑器的集成行为，
包括扩展激活、命令注册、事件处理等。
"""
import pytest
import asyncio
from typing import AsyncGenerator
from unittest.mock import AsyncMock, MagicMock
@pytest.fixture
async def vscode_extension() -> AsyncGenerator:
"""创建 VSCode 扩展模拟实例"""
extension = MagicMock()
extension.activate = AsyncMock()
extension.deactivate = AsyncMock()
# 模拟 VSCode API
extension.vscode_api = MagicMock()
extension.vscode_api.registerCommand = MagicMock(return_value=MagicMock())
extension.vscode_api.workspace = MagicMock()
extension.vscode_api.window = MagicMock()
yield extension
await extension.deactivate()
class TestVSCodeExtensionActivation:
"""VSCode 扩展激活测试"""
@pytest.mark.asyncio
async def test_extension_activates_successfully(self, vscode_extension):
"""扩展应成功激活"""
from monkeycode.vscode.extension import activate
context = MagicMock()
result = await activate(vscode_extension, context)
assert result.status == 'ok'
assert vscode_extension.activate.called
@pytest.mark.asyncio
async def test_commands_registered_on_activation(self, vscode_extension):
"""激活时应注册所有预期命令"""
from monkeycode.vscode.extension import EXPECTED_COMMANDS
context = MagicMock()
await activate(vscode_extension, context)
registered_calls = [
call.args[0] for call in
vscode_extension.vscode_api.registerCommand.call_args_list
]
for cmd in EXPECTED_COMMANDS:
assert cmd in registered_calls, f"Command '{cmd}' not registered"
class TestEditorIntegration:
"""编辑器交互集成测试"""
@pytest.mark.asyncio
async def test_completion_triggered_on_typing(self, vscode_extension):
"""输入时应触发补全"""
from monkeycode.core import CompletionTrigger
trigger = CompletionTrigger(vscode_extension)
# 模拟打字事件
event = {
'type': 'textChange',
'document': {'languageId': 'typescript'},
'position': {'line': 0, 'column': 10},
'text': 'func',
}
should_trigger = await trigger.should_trigger(event)
assert should_trigger is True
@pytest.mark.asyncio
async def test_file_save_triggers_analysis(self, vscode_extension):
"""保存文件应触发分析"""
from monkeycode.core import FileAnalyzer
analyzer = FileAnalyzer(vscode_extension)
save_event = {
'uri': 'file:///project/src/main.ts',
'content': 'export function hello() {}',
}
analysis_result = await analyzer.analyze_on_save(save_event)
assert analysis_result is not None
assert analysis_result.language == 'typescript'
assert analysis_result.functions_count >= 1
class TestMultiFileContext:
"""多文件上下文测试"""
@pytest.mark.asyncio
async def test_cross_file_reference_resolution(self, vscode_extension):
"""跨文件引用解析"""
from monkeycode.context import ContextBuilder
builder = ContextBuilder(vscode_extension)
# 模拟打开多个文件
files = {
'types.ts': 'interface User { name: string; age: number; }',
'utils.ts': 'function createUser(): User { return { name: "", age: 0 }; }',
'main.ts': 'const user = createUser();',
}
context = await builder.build_context(files, active_file='main.ts')
# 应该包含跨文件的类型信息
assert 'User' in context.type_definitions
assert 'createUser' in context.function_signatures
@pytest.mark.asyncio
async def test_context_size_limit(self, vscode_extension):
"""上下文大小限制"""
from monkeycode.context import ContextBuilder
builder = ContextBuilder(
vscode_extension,
max_context_tokens=4000
)
# 创建一个大文件
large_content = '\n'.join([f'// line {i}' for i in range(10000)])
files = {'large.ts': large_content}
context = await builder.build_context(files, active_file='large.ts')
# 上下文应在限制范围内
assert context.token_count <= 4500 # 允许一定余量
# ===== monkeycode/tests/integration/test_ai_provider_integration.py =====
"""
MonkeyCode AI Provider 集成测试
测试各种 AI 后端的集成：
- OpenAI GPT 系列
- Anthropic Claude 系列
- 开源模型 (Ollama/vLLM)
- Azure OpenAI
"""
import pytest
import httpx
from unittest.mock import patch, AsyncMock
class TestOpenAIIntegration:
"""OpenAI API 集成测试"""
@pytest.fixture
def openai_config(self):
return {
'provider': 'openai',
'api_key': 'test-key',
'model': 'gpt-4',
'base_url': 'https://api.openai.com/v1',
'max_tokens': 256,
'temperature': 0.2,
}
@pytest.mark.asyncio
@patch('httpx.AsyncClient.post')
async def test_successful_completion(self, mock_post, openai_config):
"""成功的补全请求"""
from monkeycode.ai.providers.openai import OpenAIProvider
mock_response = AsyncMock()
mock_response.status_code = 200
mock_response.json.return_value = {
'id': 'cmpl-test',
'choices': [{
'index': 0,
'text': 'return a + b;',
'finish_reason': 'stop',
}],
'usage': {'prompt_tokens': 50, 'completion_tokens': 8},
}
mock_post.return_value = mock_response
provider = OpenAIProvider(openai_config)
result = await provider.complete(
prompt='function sum(a, b) {',
language='javascript'
)
assert result.text == 'return a + b;'
assert result.finish_reason == 'stop'
assert result.usage.prompt_tokens == 50
@pytest.mark.asyncio
@patch('httpx.AsyncClient.post')
async def test_error_handling_429(self, mock_post, openai_config):
"""Rate Limit 错误处理"""
from monkeycode.ai.providers.openai import OpenAIProvider
from monkeycode.ai.errors import RateLimitError
mock_response = AsyncMock()
mock_response.status_code = 429
mock_response.headers = {'Retry-After': '5'}
mock_post.return_value = mock_response
provider = OpenAIProvider(openai_config)
with pytest.raises(RateLimitError) as exc_info:
await provider.complete(prompt='test')
assert exc_info.value.retry_after == 5
class TestOllamaIntegration:
"""本地 Ollama 模型集成测试"""
@pytest.fixture
def ollama_config(self):
return {
'provider': 'ollama',
'base_url': 'http://localhost:11434',
'model': 'codellama:13b',
'max_tokens': 512,
}
@pytest.mark.asyncio
@patch('httpx.AsyncClient.post')
async def test_local_model_connection(self, mock_post, ollama_config):
"""本地模型连接测试"""
from monkeycode.ai.providers.ollama import OllamaProvider
mock_response = AsyncMock()
mock_response.status_code = 200
mock_response.json.return_value = {
'response': 'console.log("Hello");',
'done': True,
}
mock_post.return_value = mock_response
provider = OllamaProvider(ollama_config)
# 测试连接
is_connected = await provider.health_check()
assert is_connected is True
# 测试补全
result = await provider.complete(prompt='// Log hello')
assert 'console.log' in result.text
第三层：AI 效果评估 (AI Effectiveness Evaluation)
# ===== monkeycode/tests/evaluation/benchmark_runner.py =====
"""
MonkeyCode AI 效果评估框架
使用标准化的 benchmark 数据集评估 AI 补全质量。
支持多种评估指标：
- Exact Match: 完全匹配率
- Edit Similarity: 编辑相似度 (Levenshtein)
- BLEU/ROUGE: N-gram 重叠度
- CodeBLEU: 代码专用 BLEU 变体
- Executable: 可执行通过率
- Human Preference: 人工偏好评分 (抽样)
"""
import json
import asyncio
import statistics
from dataclasses import dataclass, field
from typing import List, Dict, Optional, Tuple
from pathlib import Path
from datetime import datetime
@dataclass
class EvaluationSample:
"""单个评估样本"""
sample_id: str
file_path: str
language: str
prefix: str # 光标前的代码
suffix: str # 光标后的代码
expected: List[str] # 一个或多个期望的补全结果
metadata: Dict = field(default_factory=dict)
@dataclass
class PredictionResult:
"""预测结果"""
sample_id: str
predictions: List[str] # 模型生成的多个候选
latency_ms: float
tokens_used: int
model_name: str
timestamp: float
@dataclass
class MetricResult:
"""指标结果"""
metric_name: str
value: float
confidence_interval: Tuple[float, float]
sample_count: int
details: Dict = field(default_factory=dict)
class BenchmarkRunner:
"""Benchmark 运行器"""
def __init__(
self,
dataset_path: Path,
output_dir: Path,
num_predictions: int = 5, # 生成多少个候选
max_samples: Optional[int] = None,
):
self.dataset_path = dataset_path
self.output_dir = output_dir
self.num_predictions = num_predictions
self.max_samples = max_samples
self.results: List[PredictionResult] = []
self.output_dir.mkdir(parents=True, exist_ok=True)
def load_dataset(self) -> List[EvaluationSample]:
"""加载 benchmark 数据集"""
samples = []
# 支持多种数据集格式
if self.dataset_path.suffix == '.jsonl':
with open(self.dataset_path) as f:
for line in f:
data = json.loads(line.strip())
samples.append(EvaluationSample(**data))
elif self.dataset_path.is_dir():
# 目录模式：每个 .json 文件是一个样本
for json_file in self.dataset_path.glob('*.json'):
with open(json_file) as f:
data = json.load(f)
samples.append(EvaluationSample(**data))
if self.max_samples:
samples = samples[:self.max_samples]
print(f"📊 Loaded {len(samples)} evaluation samples")
return samples
async def run_evaluation(self, engine) -> List[PredictionResult]:
"""运行评估"""
samples = self.load_dataset()
results = []
total = len(samples)
for i, sample in enumerate(samples):
try:
start_time = asyncio.get_event_loop().time()
# 调用引擎生成补全
response = await engine.complete({
'filePath': sample.file_path,
'language': sample.language,
'position': self._find_position(sample.prefix),
'content': sample.prefix + sample.suffix,
'numPredictions': self.num_predictions,
})
latency = (asyncio.get_event_loop().time() - start_time) * 1000
result = PredictionResult(
sample_id=sample.sample_id,
predictions=[c.text for c in response.completions],
latency_ms=latency,
tokens_used=response.usage.total_tokens if response.usage else 0,
model_name=response.model or 'unknown',
timestamp=datetime.now().timestamp(),
)
results.append(result)
# 进度显示
if (i + 1) % 10 == 0 or i == total - 1:
print(f" Progress: {i+1}/{total} ({(i+1)/total*100:.1f}%)")
except Exception as e:
print(f" ⚠️ Error on sample {sample.sample_id}: {e}")
# 记录空结果
results.append(PredictionResult(
sample_id=sample.sample_id,
predictions=[],
latency_ms=-1,
tokens_used=0,
model_name='error',
timestamp=datetime.now().timestamp(),
))
self.results = results
return results
def compute_metrics(self) -> List[MetricResult]:
"""计算所有评估指标"""
metrics = []
# 1. Exact Match Rate (完全匹配率)
em_rate = self._compute_exact_match()
metrics.append(em_rate)
# 2. Edit Similarity (编辑相似度)
es_metric = self._compute_edit_similarity()
metrics.append(es_metric)
# 3. Accept@k (前 k 个候选中至少有一个被接受)
accept_metrics = self._compute_accept_at_k(k_values=[1, 3, 5])
metrics.extend(accept_metrics)
# 4. Latency Metrics (延迟指标)
latency_metrics = self._compute_latency_metrics()
metrics.extend(latency_metrics)
# 5. Per-Language Breakdown (分语言统计)
lang_metrics = self._compute_per_language_metrics()
metrics.extend(lang_metrics)
return metrics
def _compute_exact_match(self) -> MetricResult:
"""计算完全匹配率"""
matches = 0
total = len(self.results)
for result in self.results:
# 找到对应的样本
sample = next(
(s for s in self.load_dataset() if s.sample_id == result.sample_id),
None
)
if not sample or not result.predictions:
continue
# 检查是否有预测完全匹配任何期望结果
for pred in result.predictions:
if pred.strip() in [e.strip() for e in sample.expected]:
matches += 1
break
rate = matches / total if total > 0 else 0
return MetricResult(
metric_name='ExactMatch',
value=round(rate * 100, 2),
confidence_interval=self._bootstrap_ci(
lambda r: any(p.strip() in s.expected for p in r.predictions
for s in [next(x for x in self.load_dataset()
if x.sample_id == r.sample_id)]
if r.predictions),
n_bootstrap=1000
),
sample_count=total,
)
def _compute_edit_similarity(self) -> MetricResult:
"""计算编辑相似度 (归一化 Levenshtein)"""
import difflib
similarities = []
for result in self.results:
if not result.predictions:
continue
sample = next(
(s for s in self.load_dataset() if s.sample_id == result.sample_id),
None
)
if not sample:
continue
best_sim = 0
for pred in result.predictions[:1]: # 取最佳预测
for expected in sample.expected:
sim = difflib.SequenceMatcher(None, pred.strip(), expected.strip()).ratio()
best_sim = max(best_sim, sim)
similarities.append(best_sim)
avg_sim = statistics.mean(similarities) if similarities else 0
return MetricResult(
metric_name='EditSimilarity',
value=round(avg_sim * 100, 2),
confidence_interval=(
round(avg_sim * 100 - 2, 2),
round(avg_sim * 100 + 2, 2),
),
sample_count=len(similarities),
)
def _compute_accept_at_k(self, k_values=[1, 3, 5]) -> List[MetricResult]:
"""计算 Accept@k 指标"""
metrics = []
for k in k_values:
accepts = 0
total = len(self.results)
for result in self.results:
if not result.predictions:
continue
sample = next(
(s for s in self.load_dataset() if s.sample_id == result.sample_id),
None
)
if not sample:
continue
top_k = result.predictions[:k]
for pred in top_k:
if pred.strip() in [e.strip() for e in sample.expected]:
accepts += 1
break
rate = accepts / total if total > 0 else 0
metrics.append(MetricResult(
metric_name=f'Accept@{k}',
value=round(rate * 100, 2),
confidence_interval=(round(rate*100-1, 2), round(rate*100+1, 2)),
sample_count=total,
))
return metrics
def _compute_latency_metrics(self) -> List[MetricResult]:
"""计算延迟指标"""
latencies = [r.latency_ms for r in self.results if r.latency_ms > 0]
if not latencies:
return [
MetricResult('Latency_P50', 0, (0, 0), 0),
MetricResult('Latency_P99', 0, (0, 0), 0),
]
sorted_lat = sorted(latencies)
p50 = sorted_lat[int(len(sorted_lat) * 0.5)]
p99 = sorted_lat[min(int(len(sorted_lat) * 0.99), len(sorted_lat)-1)]
mean_lat = statistics.mean(latencies)
return [
MetricResult('Latency_P50', round(p50, 1), (p50-20, p50+20), len(latencies)),
MetricResult('Latency_P99', round(p99, 1), (p99-50, p99+50), len(latencies)),
MetricResult('Latency_Mean', round(mean_lat, 1), (mean_lat-10, mean_lat+10), len(latencies)),
]
def generate_report(self) -> str:
"""生成评估报告"""
metrics = self.compute_metrics()
report_lines = [
'# MonkeyCode AI 效果评估报告',
f'\n📅 评估时间: {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}',
f'📊 样本总数: {len(self.results)}',
'',
'## 📈 核心指标',
'',
'| 指标 | 数值 | 置信区间 | 样本数 |',
'|------|------|---------|--------|',
]
for m in metrics:
ci_str = f"[{m.confidence_interval[0]}, {m.confidence_interval[1]}]"
report_lines.append(
f"| {m.metric_name} | **{m.value}** | {ci_str} | {m.sample_count} |"
)
report_lines.extend([
'',
'## 📋 详细分析',
'',
'### 各语言表现',
'',
])
# 按语言分组统计
lang_stats = {}
for m in metrics:
if m.metric_name.startswith('Lang_'):
lang = m.metric_name.replace('Lang_', '')
lang_stats[lang] = m.value
for lang, score in sorted(lang_stats.items(), key=lambda x: -x[1]):
bar_len = int(score / 5)
bar = '█' * bar_len + '░' * (20 - bar_len)
report_lines.append(f"| {lang:<12} | {bar} | {score:.1f}% |")
report_content = '\n'.join(report_lines)
# 保存报告
report_path = self.output_dir / f'report_{datetime.now().strftime("%Y%m%d_%H%M%S")}.md'
report_path.write_text(report_content)
print(f"\n✅ Report saved to: {report_path}")
return report_content
@staticmethod
def _find_position(prefix: str) -> dict:
"""从 prefix 中推断光标位置"""
lines = prefix.split('\n')
return {
'line': len(lines) - 1,
'column': len(lines[-1]),
}
def _bootstrap_ci(self, scoring_fn, n_bootstrap=1000) -> tuple:
"""Bootstrap 置信区间"""
import random
scores = [scoring_fn(r) for r in self.results if r.predictions]
if not scores:
return (0, 0)
bootstrapped = []
for _ in range(n_bootstrap):
sample = [scores[random.randint(0, len(scores)-1)] for _ in scores]
bootstrapped.append(statistics.mean(sample))
bootstrapped.sort()
lower = bootstrapped[int(n_bootstrap * 0.025)]
upper = bootstrapped[int(n_bootstrap * 0.975)]
return (round(lower, 2), round(upper, 2))
def _compute_per_language_metrics(self) -> List[MetricResult]:
"""分语言计算指标"""
# 收集所有样本的语言信息
samples_by_lang = {}
for sample in self.load_dataset():
lang = sample.language
if lang not in samples_by_lang:
samples_by_lang[lang] = []
samples_by_lang[lang].append(sample.sample_id)
metrics = []
for lang, sample_ids in samples_by_lang.items():
lang_results = [r for r in self.results if r.sample_id in sample_ids]
if not lang_results:
continue
matches = sum(
1 for r in lang_results if r.predictions and any(
p.strip() in s.expected
for p in r.predictions[:1]
for s in [next(x for x in self.load_dataset()
if x.sample_id == r.sample_id)]
)
)
rate = matches / len(lang_results) * 100
metrics.append(MetricResult(
metric_name=f'Lang_{lang}',
value=round(rate, 2),
confidence_interval=(round(rate-3, 2), round(rate+3, 2)),
sample_count=len(lang_results),
))
return metrics
# 使用示例
if __name__ == '__main__':
runner = BenchmarkRunner(
dataset_path=Path('./benchmark_datasets/human_eval'),
output_dir=Path('./eval_results'),
num_predictions=5,
max_samples=None, # 使用全部样本
)
import asyncio
from monkeycode.core.engine import CompletionEngine
engine = CompletionEngine.from_config('./config/default.yaml')
# 运行评估
asyncio.run(runner.run_evaluation(engine))
# 生成报告
report = runner.generate_report()
print(report)
三、自动化 CI/CD 测试流水线
3.1 GitHub Actions 工作流
# ===== .github/workflows/ci.yml =====
name: MonkeyCode CI
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
schedule:
# 每天 UTC 2:00 (北京时间 10:00) 运行 nightly 测试
- cron: '0 2 * * *'
env:
NODE_VERSION: '20'
PYTHON_VERSION: '3.11'
jobs:
# === Job 1: Lint & 类型检查 ===
lint:
name: 🔍 Lint & Type Check
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: ESLint
run: npm run lint
- name: TypeScript type check
run: npm run typecheck
- name: Format check (Prettier)
run: npm run format:check
# === Job 2: 单元测试 ===
unit-tests:
name: 🧪 Unit Tests
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
node-version: [18, 20, 22]
steps:
- uses: actions/checkout@v4
- name: Setup Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run unit tests
run: npm run test:unit -- --coverage --ci
- name: Upload coverage
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./coverage/lcov.info
fail_ci_if_error: false
# === Job 3: 集成测试 ===
integration-tests:
name: 🔗 Integration Tests
runs-on: ubuntu-latest
needs: [lint, unit-tests]
services:
ollama:
image: ollama/ollama:latest
ports:
- 11434:11434
redis:
image: redis:7-alpine
ports:
- 6379:6379
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install dependencies
run: |
npm ci
pip install -r tests/requirements.txt
- name: Pull Ollama models
run: docker exec ollama ollama pull codellama:7b
- name: Run integration tests
run: npm run test:integration
env:
AI_PROVIDER: ollama
OLLAMA_BASE_URL: http://localhost:11434
REDIS_URL: redis://localhost:6379
# === Job 4: AI 效果评估 (Nightly) ===
ai-evaluation:
name: 🤖 AI Evaluation
runs-on: ubuntu-latest
needs: [integration-tests]
if: github.event_name == 'schedule' || github.event_name == 'push'
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install dependencies
run: pip install -r tests/eval_requirements.txt
- name: Download benchmark datasets
run: |
mkdir -p benchmark_datasets
curl -L -o benchmark_datasets/human_eval.zip \
https://storage.googleapis.com/monkeyCode-benchmarks/latest.zip
unzip benchmark_datasets/human_eval.zip -d benchmark_datasets/
- name: Run AI evaluation
run: |
python tests/evaluation/run_benchmark.py \
--dataset benchmark_datasets/human_eval \
--output eval_results \
--model gpt-4 \
--num-samples 500
env:
OPENAI_API_KEY: ${{ secrets.EVAL_OPENAI_KEY }}
- name: Upload evaluation results
uses: actions/upload-artifact@v4
with:
name: ai-evaluation-results
path: eval_results/
retention-days: 90
- name: Check quality gates
run: |
python tests/eval/check_quality_gates.py \
--results eval_results/report_latest.md \
--threshold-accept1 60 \
--threshold-latency-p99 1000
# === Job 5: 安全扫描 ===
security:
name: 🔒 Security Scan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
scan-type: 'fs'
scan-ref: '.'
severity: 'CRITICAL,HIGH'
exit-code: '1'
- name: Run npm audit
run: npm audit --audit-level=high
- name: CodeQL Analysis
uses: github/codeql-action/analyze@v3
with:
languages: javascript, typescript, python
# === Job 6: E2E 测试 ===
e2e-tests:
name: 🌐 E2E Tests
runs-on: ubuntu-latest
needs: [lint, unit-tests]
strategy:
matrix:
browser: [chromium, firefox]
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Install Playwright browsers
run: npx playwright install --with-deps ${{ matrix.browser }}
- name: Run E2E tests
run: npm run test:e2e
env:
PLAYWRIGHT_BROWSER: ${{ matrix.browser }}
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-results-${{ matrix.browser }}
path: test-results/
retention-days: 7
四、质量门禁 (Quality Gates)
4.1 分级质量标准
# ===== monkeycode/quality-gates.yaml =====
quality_gates:
# === 必须通过的门槛 (Blocking) ===
must_pass:
- name: "单元测试通过率"
metric: unit_test_pass_rate
operator: ">="
threshold: 100
message: "所有单元测试必须通过"
- name: "无新增编译错误"
metric: compile_errors
operator: "=="
threshold: 0
message: "不允许引入新的编译错误"
- name: "无 CRITICAL 安全漏洞"
metric: critical_vulnerabilities
operator: "=="
threshold: 0
message: "必须修复所有 CRITICAL 级别安全漏洞"
- name: "类型检查通过"
metric: typescript_errors
operator: "=="
threshold: 0
message: "TypeScript 类型检查必须通过"
# === 应该达到的标准 (Warning) ===
should_pass:
- name: "代码覆盖率"
metric: code_coverage
operator: ">="
threshold: 80
message: "代码覆盖率建议 ≥ 80%"
- name: "Accept@1 准确率"
metric: accept_at_1
operator: ">="
threshold: 60
message: "AI 补全首选项接受率建议 ≥ 60%"
- name: "P99 延迟"
metric: latency_p99_ms
operator: "<="
threshold: 1000
message: "P99 延迟建议 ≤ 1000ms"
- name: "崩溃率"
metric: crash_rate_per_1000
operator: "<="
threshold: 1
message: "每千次调用崩溃率建议 ≤ 0.1%"
- name: "ESLint 错误数"
metric: eslint_errors
operator: "=="
threshold: 0
message: "不应有 ESLint error 级别问题"
# === 优秀标准 (Stretch Goals) ===
excellent:
- name: "Accept@5 准确率"
metric: accept_at_5
operator: ">="
threshold: 88
message: "优秀: 前5项接受率 ≥ 88%"
- name: "P50 延迟"
metric: latency_p50_ms
operator: "<="
threshold: 200
message: "优秀: 中位延迟 ≤ 200ms"
- name: "代码覆盖率"
metric: code_coverage
operator: ">="
threshold: 90
message: "优秀: 覆盖率 ≥ 90%"
4.2 质量仪表盘关键指标
┌─────────────────────────────────────────────────────────────┐
│ 📊 MonkeyCode 质量仪表盘 (实时) │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ ✅ 单元测试 │ │ 🟢 覆盖率 │ │ 🟢 ESLint │ │
│ │ 通过率: 100% │ │ 84.2% │ │ 0 errors │ │
│ │ 5234/5234 │ │ 目标: 80%+ │ │ 3 warnings │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ 🤖 AI 效果指标 (最近 24h) │ │
│ │ │ │
│ │ Accept@1: ████████████████░░░░ 67.3% 🟢 目标≥60% │ │
│ │ Accept@5: ████████████████████░ 89.1% 🟢 目标≥85% │ │
│ │ ExactMatch: ██████████░░░░░░░░░░ 42.8% │ │
│ │ EditSim: ████████████████████░ 87.5% │ │
│ │ │ │
│ │ P50 延迟: ████░░░░░░░░░░░░░░░░░ 185ms 🟢 ≤200ms │ │
│ │ P99 延迟: ████████████████████░ 782ms 🟢 ≤1000ms │ │
│ │ 错误率: ██░░░░░░░░░░░░░░░░░░░░ 0.03% 🟢 ≤0.1% │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ 🔒 安全状态 │ │
│ │ CRITICAL: 0 HIGH: 2 MEDIUM: 12 LOW: 28 │ │
│ │ 上次扫描: 2 小时前 │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
╚═══════════════════════════════════════════════════════════╝
五、开源社区的参与方式
5.1 如何贡献测试用例
## 为 MonkeyCode 贡献测试用例指南
### 为什么需要你的贡献？
MonkeyCode 的 AI 能力在不断进化，我们需要更多真实场景的测试样本来：
- 发现在特定语言/框架下的边界情况
- 提升对罕见编程模式的识别能力
- 确保 AI 输出的多样性和准确性
### 如何提交测试用例？
#### 方式一：提交 Benchmark 样本
1. Fork [monkeycode-ai/benchmark-datasets](https://github.com/monkeycode-ai/benchmark-datasets) 仓库
2. 在 `contributions/` 目录下创建 JSON 文件：
```json
{
"sample_id": "contrib-yourname-001",
"file_path": "src/components/UserForm.tsx",
"language": "typescript",
"prefix": "interface UserFormProps {\n onSubmit: (data: UserData) => void;\n initialData?: ",
"suffix": ";\n}\n\nexport function UserForm({ onSubmit, initialData }: UserFormProps) {\n return <form>",
"expected": [
"Partial<UserData>;",
"UserData | undefined;"
],
"difficulty": "medium",
"tags": ["react", "typescript", "interfaces"],
"contributor": "your-github-username",
"notes": "Testing React component prop inference with optional generic type"
}
- 提交 PR，我们会审核并合入到下一个版本
方式二：报告 AI 输出问题
如果你发现 MonkeyCode 的输出有问题，请通过以下方式反馈：
-
GitHub Issue (推荐):
- 前往 Issues
- 选择模板
AI Output Issue - 填写触发条件、期望输出、实际输出
- 如果方便的话，附上截图或录屏
-
内置反馈按钮:
- 在 IDE 中点击 AI 补全结果旁的 👎 按钮
- 选择问题类型并提交
- 反馈会自动包含上下文信息
贡献者激励
- 🏆 Top Contributor 月榜: 每月评选最多高质量样本的贡献者
- 📛 荣誉墙: 在 README 和官网展示活跃贡献者
- 🎁 限量周边: MonkeyCode 定制 T恤、贴纸等
- 💼 优先体验: 新功能内测资格
---
## 六、经验总结与未来方向
### 6.1 核心经验总结
┌─────────────────────────────────────────────────────────────┐
│ MonkeyCode 测试策略核心经验 │
│ │
│ 1. 测试金字塔依然适用，但顶层需要重新定义 │
│ AI 的"端到端测试"不是验证功能流程，而是验证输出质量 │
│ │
│ 2. 自动化评估 + 人工抽检相结合 │
│ 自动化 benchmark 保证基线，人工评审发现微妙质量问题 │
│ │
│ 3. 回归测试是 AI 产品最大的挑战 │
│ 模型更新可能导致已有测试全部失效，需要建立鲁棒的评估集 │
│ │
│ 4. 性能和质量是不可分割的 │
│ 更好的模型通常意味着更高的延迟，需要找到最优平衡点 │
│ │
│ 5. 社区参与是提升测试覆盖率的最佳途径 │
│ 真实用户的场景比任何合成数据都有价值 │
│ │
│ 6. 安全测试不能事后补救 │
│ AI 输入/输出的安全校验必须在设计阶段就纳入 │
│ │
└─────────────────────────────────────────────────────────────┘
### 6.2 未来规划
```yaml
future_testing_plans:
2026_q3:
- "引入 Agent 评估: 用更强模型自动评估弱模型的输出质量"
- "扩展多语言 benchmark: 覆盖 Rust/Go/Swift/Kotlin 等"
- "实时 A/B 测试框架: 在生产环境对比不同 Prompt 策略"
2026_q4:
- "用户行为驱动的测试: 从真实使用数据自动生成回归测试"
- "对抗性测试: 自动构造 edge case 攻击 AI 输出"
- "可解释性评估: 评估 AI 输出的可理解程度"
2027_h1:
- "联邦学习下的质量评估: 保护隐私的分布式效果评估"
- "跨 IDE 一致性测试: 确保不同编辑器下行为一致"
- "自动化质量回滚: 当指标下降时自动阻止发布"
结语
"质量不是测试出来的，而是设计和构建出来的。但对于 AI 产品来说，持续的测试和评估是保持质量的唯一途径。"
MonkeyCode 的测试体系仍在不断演进中。我们相信，通过开放透明的质量标准和社区协作的力量，我们可以构建出一个既智能又可靠的 AI 编程助手。
如果你对 MonkeyCode 的测试策略感兴趣，或者想为我们的 benchmark 数据集贡献力量，欢迎通过 GitHub 与我们交流！
💡 相关资源:
- 🧪 Benchmark 数据集
- 📊 质量仪表盘
- 🐛 提交 Bug
- 💬 测试讨论区
- 📖 测试开发文档
MonkeyCode — 用严格的质量标准，守护每一次智能补全的可靠性。 🐵🧪✨
浙公网安备 33010602011771号