Weekly Review Prompt

# Weekly Calibration Review Prompt

Run this prompt against your AI once a week. It reads `calibration_log.jsonl` and produces a calibration report.

---

**Prompt:**

> Read `calibration_log.jsonl`. For each unique `component`, group the entries by stated confidence bucket:
> - Low: 0.5–0.7
> - Medium: 0.7–0.9
> - High: 0.9+
>
> For each (component, bucket), compute the actual correctness rate (correct / total).
>
> Produce a markdown table with columns: Component, Bucket, Sample Size, Stated Range, Actual Rate, Delta.
>
> After the table, write 2–3 sentences identifying:
> 1. Which components are well-calibrated (delta within 10 points)
> 2. Which components are overconfident (stated >> actual) — these should be down-weighted
> 3. Which components are underconfident (actual >> stated) — these can be trusted more
>
> If any component has fewer than 10 samples in a bucket, flag it as "insufficient data."
>
> Save the report to `calibration_reports/YYYY-MM-DD.md`.

---

## What to Do With the Report

- **Well-calibrated components** — no action needed
- **Overconfident components** — add a note to your system prompt: "When using `<component>`, state confidence levels that are 10–20 points lower than your instinct."
- **Underconfident components** — either trust their outputs more, or investigate why they're being cautious
- **Insufficient data** — keep collecting before drawing conclusions