home/productivity/prompt-eval-d-ai-evaluation

Prompt Eval D ai Evaluation

GPTClaudeDeepSeek··▲ 1,064 copies·updated 2026-07-14

prompt-eval-d-ai-evaluation.prompt

# 子项目 D:AI 自动评估 Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans. Steps use checkbox (`- [ ]`) syntax.

**Goal:** 用一个评判大模型对某评测的 item 自动判优(多臂选最佳 / 单 prompt 判好坏),把 AI 判定与人工 verdict 并存写回,展示来源与 AI 推理;评判系统指令在设置页可配。

**Architecture:** 给 `prompt_eval_items` 加并行 AI 列(迁移 025)+ 新建通用 `app_settings` 表存评判指令(默认值在代码);触发 API 派发 Celery worker,worker 逐条 pending item 调评判模型、强制 JSON 解析、写 AI 列;前端设置页编辑指令 + 列表/工作台触发与展示。

**Tech Stack:** FastAPI + SQLAlchemy + 编号迁移;Celery worker(原始 SQL)+ `llm_client`;pytest;React + TS。

**Spec:** [`docs/superpowers/specs/2026-06-15-prompt-eval-D-ai-evaluation-design.md`](../specs/2026-06-15-prompt-eval-D-ai-evaluation-design.md)

**铁律(`CLAUDE.md`)**:改 `app/models/**` 必配编号迁移;app-server 与 train-worker 互不 import(worker 用原始 SQL + 共享 `modelforge_common`);macOS worker `--pool=solo`。当前最新迁移 `024`,新增 `025`。提交以 `Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>` 结尾。

## 文件结构

| 文件 | 职责 | 任务 |
|---|---|---|
| `services/app-server/app/models/prompt_eval.py` | PromptEvalItem 加 AI 列 | T1 |
| `services/app-server/app/models/setting.py` + `models/__init__.py` | `AppSetting` | T1 |
| `services/app-server/db/migrations/025_ai_evaluation.sql` | AI 列 + app_settings 表 | T1 |
| `services/app-server/tests/test_migrations_apply.py` | ntab 21→22 | T1 |
| `services/common/modelforge_common/task_names.py` | `PROMPT_AI_EVAL_TASK` | T2 |
| `services/app-server/app/ai_eval_defaults.py` | `DEFAULT_AI_EVAL_PROMPT` | T2 |
| `services/app-server/app/services/ai_eval_service.py` | get/set prompt + dispatch | T2 |
| `services/app-server/app/celery_client.py` | `send_prompt_ai_eval_task` | T2 |
| `services/app-server/app/schemas/prompt_eval.py` + `schemas/setting.py` | ItemOut AI 字段 + AiEvaluateIn + 设置 schema | T3 |
| `services/app-server/app/api/settings.py` + `api/prompt_eval.py` + `main.py` | 设置 API + 触发 API + ItemOut 构造 | T3 |
| `services/app-server/tests/test_ai_eval.py` | service/API 测试 | T2,T3 |
| `services/train-worker/worker/prompt_ai_eval.py` + `worker/db.py` + `worker/tasks.py` | worker 评判 | T4 |
| `services/train-worker/tests/test_prompt_ai_eval_worker.py` | worker 测试 | T4 |
| `frontend/src/api/client.ts` | client | T5 |
| `frontend/src/pages/SettingsPage.tsx` + `PromptEvalsPage.tsx` | 设置区 + 触发 | T6 |
| `frontend/src/pages/PromptEvalWorkbench.tsx` | AI 结果展示 | T7 |

---

### Task 1: AI 列 + AppSetting 模型 + 迁移 025

**Files:**
- Modify: `services/app-server/app/models/prompt_eval.py`
- Create: `services/app-server/app/models/setting.py`
- Modify: `services/app-server/app/models/__init__.py`
- Create: `services/app-server/db/migrations/025_ai_evaluation.sql`
- Modify: `services/app-server/tests/test_migrations_apply.py`
- Test: `services/app-server/tests/test_ai_eval.py`

- [ ] **Step 1: 写失败测试**

Create `services/app-server/tests/test_ai_eval.py`:

when to use it

Community prompt sourced from the open-source GitHub repo LingyiChen-AI/ModelForge (Apache-2.0). A "Prompt Eval D ai Evaluation" style prompt — adapt the placeholders and specifics to your task. Imported as-is and not independently retested here, so check the output before relying on it.