Llm as Judge Prompts

# LLM-as-Judge Evaluation Prompts

- **Use when**: Setting up automated quality evaluation, or gating deployments on output quality
- **Time**: 1-2 hours to adapt prompts
- **Outcome**: Production prompts for helpfulness, accuracy, safety evaluation
- **Related**: [eval-gate.yml](eval-gate.yml) ・ [Before/After Patterns](before-after-patterns.md)

---

Production-grade prompts for evaluating LLM outputs. These are not academic exercises - they're the prompts used to gate deployments.

---

## Why LLM-as-Judge

| Approach | Cost | Latency | Catches |
|----------|------|---------|---------|
| Golden set (exact match) | Free | Fast | Syntax regressions |
| Embedding similarity | Low | Fast | Semantic drift |
| **LLM-as-Judge** | Medium | Slow | Quality, nuance, reasoning |
| Human review | High | Very slow | Everything |

**Use LLM-as-Judge when:**
- You need to evaluate subjective quality (helpfulness, clarity, tone)
- Golden sets are too rigid for your output format
- You're gating a release and need a quality signal

**Don't use it when:**
- You need deterministic results (it's stochastic)
- You're evaluating factual accuracy (judges hallucinate too)
- Latency matters (adds 1-5s per eval)

---

## The Calibration Problem

LLM judges have systematic biases:
- **Verbosity bias**: Longer responses score higher
- **Position bias**: First option in comparisons scores higher
- **Self-enhancement**: Models rate their own outputs higher
- **Sycophancy**: Models agree with the human's apparent preference

**Fix**: Calibrate against human labels. Run 100+ examples with human ratings, measure correlation, adjust prompts until R² > 0.7.

---

## Prompt 1: Helpfulness Evaluation

**When to use**: Evaluating if the response actually solves the user's problem.