home/productivity/prompt-eval-c1-eval-engine

Prompt Eval C1 Eval Engine

GPTClaudeDeepSeek··▲ 812 copies·updated 2026-07-14

prompt-eval-c1-eval-engine.prompt

# 子项目 C1:Prompt 评测引擎 + 三种评测 + 数据模型 Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** 发起三种 Prompt 评测(多 prompt 盲测 / 多模型盲测 / 单 prompt 版本对比),worker 调用大模型把测试集每行 × 每臂批量产出结果(状态=待评估),为 C2 的人工评估备好数据。

**Architecture:** 统一数据模型 run / arm / item / output;app-server 校验(prompt 参数 ⊆ 测试集字段)+ 建 run/arms + 派发 Celery;train-worker 渲染模板、调 `llm_client`(A)、写 outputs、报进度;`prompt_template.render`(补 B 遗留)。

**Tech Stack:** FastAPI + SQLAlchemy 2.0 + 编号 SQL 迁移;Celery worker(原始 SQL);httpx via llm_client;pytest;React + TS + Vite。

**Spec:** [`docs/superpowers/specs/2026-06-15-prompt-eval-C1-eval-engine-design.md`](../specs/2026-06-15-prompt-eval-C1-eval-engine-design.md)

**铁律(`CLAUDE.md`)**:改 `app/models/**` 必配【下一个编号】幂等迁移;改 RBAC 时 `bootstrap.py` 与迁移两处一起改;**app-server 与 train-worker 互不 import**(worker 用原始 SQL + 共享 `modelforge_common`);macOS worker `--pool=solo`。当前最新迁移 `020`,本计划新增 `021`、`022`。提交以 `Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>` 结尾。

## 文件结构

| 文件 | 职责 | 任务 |
|---|---|---|
| `services/common/modelforge_common/prompt_template.py` | 加 `render()` | T1 |
| `services/common/modelforge_common/task_names.py` | 加 `PROMPT_EVAL_TASK` | T1 |
| `services/common/tests/test_prompt_template.py` | render 单测 | T1 |
| `services/app-server/app/models/prompt_eval.py` | 4 个模型 | T2 |
| `services/app-server/app/models/__init__.py` | 注册 | T2 |
| `services/app-server/db/migrations/021_prompt_evals.sql` | 4 表 | T2 |
| `services/app-server/app/bootstrap.py` + `db/migrations/022_prompteval_perms.sql` | 权限 | T3 |
| `tests/test_bootstrap.py`、`tests/test_migrations_apply.py` | 计数(21/21) | T3 |
| `services/app-server/app/schemas/prompt_eval.py` | schema | T4 |
| `services/app-server/app/services/prompt_eval_service.py` + `app/celery_client.py` | 校验+建模+派发 | T5 |
| `services/app-server/app/api/prompt_eval.py` + `app/main.py` | `/prompt-evals` + options | T6 |
| `services/app-server/tests/test_prompt_eval.py` | service/API 测试 | T2,T5,T6 |
| `services/train-worker/worker/prompt_eval.py` + `worker/db.py` + `worker/tasks.py` | worker 引擎 | T7 |
| `services/train-worker/tests/test_prompt_eval_worker.py` | worker 测试 | T7 |
| `frontend/src/api/client.ts` | client | T8 |
| `frontend/src/pages/PromptEvalsPage.tsx` + `App.tsx` + `AppShell.tsx` | 页面 | T9 |

---

### Task 1: `render()` + task name(common)

**Files:**
- Modify: `services/common/modelforge_common/prompt_template.py`
- Modify: `services/common/modelforge_common/task_names.py`
- Test: `services/common/tests/test_prompt_template.py` (APPEND)

- [ ] **Step 1: 追加失败测试**

Append to `services/common/tests/test_prompt_template.py`:

when to use it

Community prompt sourced from the open-source GitHub repo LingyiChen-AI/ModelForge (Apache-2.0). A "Prompt Eval C1 Eval Engine" style prompt — adapt the placeholders and specifics to your task. Imported as-is and not independently retested here, so check the output before relying on it.