Why Not Promptfoo

GPTClaudeDeepSeek··▲ 1,255 copies·updated 2026-07-14

why-not-promptfoo.prompt

# Why not promptfoo?

[promptfoo](https://github.com/promptfoo/promptfoo) is the most popular open-source LLM eval tool today (~5k stars, MIT, very active). When I started building eval-harness in May 2026 I evaluated promptfoo first. I still recommend it for most teams.

This page is the honest answer to **"why does eval-harness exist if promptfoo already does this?"** It's organized around four things eval-harness does that promptfoo doesn't, and three things promptfoo does that eval-harness doesn't.

> **One-line summary:** promptfoo is a great _eval framework_. eval-harness is a focused _regression-detection harness_. They solve different problems and compose well.

## What eval-harness does that promptfoo doesn't

### 1. Failure attribution (the killer feature)

When a promptfoo test fails, you get `expected`, `actual`, and a diff. You then spend 20–60 minutes asking yourself:

- _Did my prompt change?_
- _Did the model change under me?_
- _Did the fixture rot?_
- _Is this flaky?_

eval-harness ships a 4-class attribution decision tree that answers that question deterministically using SHA fields captured at baseline + at run time:

when to use it

Community prompt sourced from the open-source GitHub repo nano-step/eval-harness (MIT). A "Why Not Promptfoo" style prompt — adapt the placeholders and specifics to your task. Imported as-is and not independently retested here, so check the output before relying on it.