Instruction Tuned Models

# Lesson 2 — Instruction-Tuned Models

## 1. Concept

Instruction tuning (IT) is supervised fine-tuning on (instruction, response) pairs, distinct in purpose from preference alignment: IT teaches format-following and task-following behavior, while alignment (RLHF/DPO) corrects values and refusal behavior. Instruction tuning on a well-established task-specific instruction set has been shown to inject domain-specific knowledge into general LLMs, transferring them into domain experts (e.g., in medicine, law, and finance), whereas alignment tuning separately corrects unexpected model behaviors to match human values and preferences [1]. Production-grade instruction-tuned models stack both, but research shows the stacking need not be strictly sequential — a unified fine-tuning approach (UFT) trained solely on instruction data, using a generalized implicit reward function, has been shown to outperform plain SFT by minimizing divergence from the pretrained model [2].

For this project, the practical distinction you're testing is **instruction-following fidelity under a rigid output contract** — something a code-compliance tool depends on completely. A model that answers "44 inches, but there are exceptions" in prose is useless to a permitting-software integration; you need `{"value": 44, "unit": "inches", "exceptions": [...], "section": "1020.2"}` every time.

## 2. Why it matters for routing decisions

Instruction-tuned models are your default for any question requiring structured output — which is most of what a building-code assistant needs to return to a downstream system (a permit-checking form, a contractor's estimating tool). The key risk you're testing: does a cheaper mid-tier instruction-tuned model hold the schema contract as reliably as the flagship foundation-tier model on numeric and jurisdiction_amendment questions?

## 3. Build increment

Add `clients/instruction_tuned_client.py`, structurally identical to Lesson 1's client but pointed at a mid-tier instruction-tuned model (GPT-4o-mini, Claude Haiku-class, or Llama-3.1-8B-Instruct), and require structured JSON output: