home/productivity/prompt-eval-c2-workbench-and-stats

Prompt Eval C2 Workbench And Stats

GPTClaudeDeepSeek··▲ 585 copies·updated 2026-07-14

prompt-eval-c2-workbench-and-stats.prompt

# 子项目 C2:盲测人工评估工作台 + 每轮统计 Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans. Steps use checkbox (`- [ ]`) syntax.

**Goal:** 让评估人对 C1 跑出的待评估输出做盲测人工评估(多臂选谁更好/单 prompt 好坏),verdict 落库,并按需算出每轮统计(各 arm 胜率与最优;单 prompt 好率与变好/变坏率)。

**Architecture:** 给 C1 的 `prompt_eval_items` 加 verdict 列(迁移 023);后端取 items 时按 `random.Random(item.id)` 打乱各臂输出做盲测匿名;verdict PATCH 落库;stats 实时聚合;前端两栏工作台 + 统计抽屉 + C1 列表入口。无 worker。

**Tech Stack:** FastAPI + SQLAlchemy + 编号 SQL 迁移;pytest;React + TS + Vite。

**Spec:** [`docs/superpowers/specs/2026-06-15-prompt-eval-C2-workbench-and-stats-design.md`](../specs/2026-06-15-prompt-eval-C2-workbench-and-stats-design.md)

**铁律(`CLAUDE.md`)**:改 `app/models/**` 必配编号迁移;改 RBAC 时 bootstrap + 迁移一起改。当前最新迁移 `022`,新增 `023`(加列)、`024`(权限)。提交以 `Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>` 结尾。

## 文件结构

| 文件 | 职责 | 任务 |
|---|---|---|
| `services/app-server/app/models/prompt_eval.py` | `PromptEvalItem` 加 verdict 列 + annotator | T1 |
| `services/app-server/db/migrations/023_prompt_eval_verdict.sql` | 加列 | T1 |
| `services/app-server/app/bootstrap.py` + `024_prompteval_annotate_perm.sql` | `prompteval:annotate` | T2 |
| `tests/test_bootstrap.py`、`tests/test_migrations_apply.py` | 计数(21 表 / 22 权限) | T2 |
| `services/app-server/app/schemas/prompt_eval.py` | `VerdictIn` + `ItemOut` 加 verdict 字段 | T3 |
| `services/app-server/app/services/prompt_eval_service.py` | `submit_verdict` | T3 |
| `services/app-server/app/services/prompt_eval_stats.py` | `stats` | T3 |
| `services/app-server/app/api/prompt_eval.py` | items 扩展 + verdict + stats | T4 |
| `services/app-server/tests/test_prompt_eval.py` | service/API 测试 | T1,T3,T4 |
| `frontend/src/api/client.ts` | client | T5 |
| `frontend/src/pages/PromptEvalWorkbench.tsx` + `App.tsx` | 工作台 | T6 |
| `frontend/src/pages/PromptEvalsPage.tsx` | 统计抽屉 + 入口按钮 | T7 |

---

### Task 1: verdict 列 + 迁移 023

**Files:**
- Modify: `services/app-server/app/models/prompt_eval.py`
- Create: `services/app-server/db/migrations/023_prompt_eval_verdict.sql`
- Test: `services/app-server/tests/test_prompt_eval.py` (APPEND)

- [ ] **Step 1: 追加失败测试**

Append to `services/app-server/tests/test_prompt_eval.py`:

when to use it

Community prompt sourced from the open-source GitHub repo LingyiChen-AI/ModelForge (Apache-2.0). A "Prompt Eval C2 Workbench And Stats" style prompt — adapt the placeholders and specifics to your task. Imported as-is and not independently retested here, so check the output before relying on it.