PromptOptimizer README

# PromptOptimizer: Automated Prompt Testing & Optimization

A production-ready system for systematic prompt engineering with A/B testing, statistical analysis, and automated prompt optimization.

## 🎯 Overview

PromptOptimizer extends LLMOps-Eval with rigorous prompt experimentation capabilities. Instead of random prompt tweaks, it provides **systematic variation strategies**, **statistical A/B testing**, and **data-driven prompt selection**.

### Key Features

- 🔄 **11 Systematic Variation Strategies** - Instruction rephrasing, few-shot selection, CoT styles, etc.
- 🧪 **A/B/n Testing Framework** - Proper experimental design with random assignment
- 📊 **Statistical Analysis** - T-tests, Mann-Whitney, effect sizes, power analysis
- 🏆 **Intelligent Selection** - Multi-criteria ranking with confidence scoring
- 📈 **Interactive Dashboard** - Streamlit UI for experiment management
- 🔁 **Reproducibility** - Seeded randomness for consistent results

## 📚 Key Concepts

### Statistical Significance

Before adopting a prompt change, ensure it's **statistically significant**:

- **P-value < α (typically 0.05)**: The improvement isn't due to chance
- **Effect Size (Cohen's d)**: Magnitude of improvement
  - 0.2 = small, 0.5 = medium, 0.8 = large
- **Statistical Power**: Probability of detecting a real effect (aim for ≥0.80)

### Multiple Comparison Correction

When testing multiple variants, correct for false positives:

- **Bonferroni**: Conservative, divides α by number of comparisons
- **Benjamini-Hochberg (FDR)**: Less conservative, controls false discovery rate
- **Holm-Bonferroni**: Step-down procedure, good balance

### Sample Size Planning

Calculate required sample size **before** experimenting: