Agent Optimization Checklist

# Agent Optimization Checklist

Use this as a short review gate before accepting an agent optimization.

## Measurement

- [ ] Baseline recorded before changes
- [ ] Candidate measured with the same harness
- [ ] Sample size and data source stated
- [ ] Median, range, and failures reported
- [ ] Full session cost counted, including retries and recoveries
- [ ] Numbers labeled as measured, estimated, or hoped

## Data

- [ ] Real production traffic used, or proxy justified
- [ ] Worst/tail cases included
- [ ] Representative slice included
- [ ] Blind holdout frozen before iteration
- [ ] Blind data provenance audited

## Correctness

- [ ] Delivery defined by what the user received
- [ ] Correctness gate passed
- [ ] Accuracy parity proven
- [ ] No silent truncation or dropped artifacts
- [ ] Behavior changes disclosed

## Engineering

- [ ] Real bottleneck traced before changes
- [ ] Deterministic levers attempted before prompt/model tuning
- [ ] Repeated reasoning moved into code where possible
- [ ] Inexpressible asks degrade to fallback, partial output, or honest refusal
- [ ] Budgets and guards keyed to identities the model cannot rewrite

## Verification

- [ ] Real serving path tested
- [ ] Streaming path tested if relevant
- [ ] Generated artifacts validated
- [ ] Reviewer claims spot-checked
- [ ] Fresh blind data used for final proof
- [ ] Kill switch and telemetry planned for rollout

## Red Flags

- "Median of successful runs"
- One prompt patch per observed failure
- Faster because output got smaller
- Test changed to match implementation
- Benchmark from an old build
- Demo prompt improved, real traffic unchanged
- Fast path returns silence when ask is unsupported
- The model can rename around a budget or cap