Prompt Injection.review

# Quality Review — prompt_injection, 2026-W27

Reviewer: Hermes agent (Quality Review stage). Checked
`reports/2026/week_27/prompt_injection.md` against
`datasets/prompt_injection/2026-W27/summary.json`,
`datasets/prompt_injection/2026-W27/scored.parquet`, and
`datasets/prompt_injection/2026-W27/raw_responses.jsonl`.

## What was checked

- **Every number in Results (§7)** — overall rate, by-variant rates, and
  the full per-model table — recomputed independently from `summary.json`
  and cross-checked against a fresh pandas aggregation of `scored.parquet`.
  All matched exactly.
- **The "by injection vector" table** — recomputed per-prompt fooled counts
  directly from `scored.parquet` (grouping by `prompt_id`, excluding error
  rows), including which specific models were fooled on each prompt.
  Matched exactly.
- **Every quoted transcript and its `run_id`** (§8) — looked up each
  `run_id` in `raw_responses.jsonl`, confirmed it belongs to the model and
  prompt the report attributes it to, and confirmed the quoted text is
  verbatim (the report's smart quotes are a markdown rendering choice, not
  a text change).
- **Reproducibility Information (§12)** — prompt set, model panel file,
  and rerun command all point to real, existing paths; chart file
  (`charts/2026-W27/prompt_injection_by_model.png`) confirmed to exist.
- **Ranking language** — scanned for implicit "best model" framing.
  Practical Implications (§9) explicitly argues *against* treating
  injection-susceptibility as a fixed per-model trait, consistent with the
  project's non-goals. The per-model results table is sorted by pass rate
  descending, which is ranking-*adjacent* presentation, but this matches
  the same convention already used in `hallucination.md`; not flagged as
  an error, noted below as a standing style question.
- **Hedging proportionality** — sample-size caveats, the "not a ranking"
  disclaimer, and the false-negative-risk limitation are all present and
  appropriately scoped to what a 4-prompt, single-replicate run can
  support.

## What was corrected

1. **Wrong error type, repeated in two places.** Results (§7) already said
   "token-budget truncation" correctly, but Limitations (§10) still said
   *"One `429` rate limit reduced `qwen/qwen3.5-9b`'s effective n to 3"* —
   the actual error (confirmed in `raw_responses.jsonl`) is a truncation
   (`TRUNCATED: hit max_tokens before producing a final answer...`), not a
   rate limit. This inconsistency meant the report contradicted itself
   between sections. Fixed to match §7 and the methodology file.
2. **Denominator error in the Executive Summary.** "Injection succeeded in
   4 of 24 scored cases (83% correct overall)" mixed the total-attempted
   count (24) with a "scored" label that should mean 23 (one row was
   excluded as an infrastructure error, per §7). 4/24 and 4/23 both round
   to a similar-looking headline figure by coincidence, which is exactly
   why it wasn't obvious on a casual read — but the correct, internally
   consistent figure is **4 of 23 (82.6%)**, matching the precise number
   already used in §7. Fixed.

Both corrections were arithmetic/consistency errors, not new claims — no
finding in the report changed as a result, only the precision of how two
already-correct results (from §7) were restated in the summary and
limitations.

## Open questions for a human reviewer

- **Style question, not an error**: should per-model results tables in
  this report series be sorted alphabetically instead of by pass rate?
  Sorting by rate reads slightly ranking-adjacent even with an explicit
  "not a ranking" disclaimer elsewhere in the report. Not changed here
  since it matches the existing `hallucination.md` convention and
  changing one report's convention without changing the others would
  itself be an inconsistency — worth a project-wide decision, not a
  per-report fix.
- **Confirm before external publication**: this report names
  `deepseek/deepseek-v4-flash` specifically as the model that fully
  abandoned its task on `injection-001`. That's accurate and directly
  supported by the data, but it's the most quotable, attention-grabbing
  claim in the report and the one most likely to be read as "this vendor's
  model is bad" despite the surrounding text explicitly framing it as
  vector-specific, not general. Worth a deliberate go/no-go on that
  framing before it's shared externally, per the project's dual-use /
  responsible-publication instinct even though this isn't a safety-harm
  case — it's a reputational one.

## Verdict

No open behavioral or scoring claims were found to be false. Two
internal-consistency/arithmetic errors were found and fixed. Recommend
approval for publication after a human confirms the framing question
above.