Prompt Eval A Llm Providers
# 子项目 A:LLM 设置页(模型供应商配置)Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** 让有 `llm:manage` 权限的用户在「设置」页配置 OpenAI 协议的大模型供应商(下挂多个 model-id)、一键测试连通性,配置全局共享供后续 Prompt 评测复用。
**Architecture:** app-server 新增 `llm_providers` / `llm_models` 两张表与 `/llm` CRUD + 测试端点;连外部大模型的逻辑抽到 `services/common` 的共享 `llm_client`(OpenAI `/chat/completions`,httpx),app-server 的测试端点同步直调、worker 后续复用;前端新增设置页。api_key 明文存库、读取掩码。
**Tech Stack:** FastAPI + SQLAlchemy 2.0 + 编号 SQL 迁移;httpx;pytest;React 19 + Vite + TS + Tailwind v4。
**Spec:** [`docs/superpowers/specs/2026-06-15-prompt-eval-A-llm-providers-design.md`](../specs/2026-06-15-prompt-eval-A-llm-providers-design.md)
**项目铁律(见 `CLAUDE.md`)**:改 `app/models/**` 必须在同一提交里加一个【下一个编号】的 `db/migrations/NNN_*.sql`(幂等);不要用 Alembic。改 RBAC 权限目录时 `app/bootstrap.py` 与编号迁移两处一起改。当前最新迁移 = `016_api_key_plaintext.sql`,本计划新增 `017`、`018`。
## 文件结构
| 文件 | 职责 | 任务 |
|---|---|---|
| `services/common/modelforge_common/llm_client.py` | OpenAI 协议 httpx 封装:`chat()` / `ChatResult` / `LLMError` | T1 |
| `services/common/pyproject.toml` | 加 `httpx` 依赖 | T1 |
| `services/common/tests/test_llm_client.py` | `llm_client` 单测 | T1 |
| `services/app-server/app/models/llm.py` | `LlmProvider` / `LlmModel` 模型 + `mask_key` | T2 |
| `services/app-server/app/models/__init__.py` | 注册新模型 | T2 |
| `services/app-server/db/migrations/017_llm_providers.sql` | 建两张表 | T2 |
| `services/app-server/app/bootstrap.py` | 加 `llm:manage` 权限 | T3 |
| `services/app-server/db/migrations/018_llm_manage_perm.sql` | 写权限种子 | T3 |
| `services/app-server/app/schemas/llm.py` | 请求/响应 schema(掩码) | T4 |
| `services/app-server/app/services/llm_provider_service.py` | provider/model CRUD + 测试 | T5 |
| `services/app-server/app/api/llm.py` | `/llm` 路由 | T6 |
| `services/app-server/app/main.py` | 注册路由 | T6 |
| `services/app-server/tests/test_llm_providers.py` | 模型/service/API 测试 | T2,T5,T6 |
| `frontend/src/api/client.ts` | LLM provider API client | T7 |
| `frontend/src/pages/SettingsPage.tsx` | 设置页 | T8 |
| `frontend/src/App.tsx` | 路由 `/settings` | T8 |
| `frontend/src/components/AppShell.tsx` | 导航「设置」入口 | T8 |
---
### Task 1: 共享 LLM 客户端 `llm_client`
**Files:**
- Create: `services/common/modelforge_common/llm_client.py`
- Modify: `services/common/pyproject.toml`
- Test: `services/common/tests/test_llm_client.py`
- [ ] **Step 1: 加 httpx 依赖**
把 `services/common/pyproject.toml` 顶部 `[project]` 段改为带依赖(其余段不动):when to use it
Community prompt sourced from the open-source GitHub repo LingyiChen-AI/ModelForge (Apache-2.0). A "Prompt Eval A Llm Providers" style prompt — adapt the placeholders and specifics to your task. Imported as-is and not independently retested here, so check the output before relying on it.
tags
productivitycommunitydeveloper
source
LingyiChen-AI/ModelForge · Apache-2.0
more in Productivity
Productivity✓ tested
Summarize a doc into decisions & actions
chief of staff who extracts what to DO, not just what was said
Productivity✓ tested
Draft a reply to a hard email
calm, direct communicator who de-escalates without caving
Productivity✓ tested
Turn a brain-dump into a weekly plan
planning coach who protects your focus, not just your calendar