Low Latency Telephony Prompting

# Prompting for small, fast models on telephony

When I first started building phone agents on ElevenLabs, the default instinct was to reach for the biggest model available. More reasoning, fewer hallucinations. That instinct is reasonable for a chatbot or an async task. It's wrong for a live phone call.

---

## The telephony latency budget

There's a well-documented perceptual threshold: around 500ms to 1.5 seconds of silence after the caller stops speaking, they start to wonder if the call dropped. By 2 seconds, some say "hello?" By 3 seconds, some hang up.

Total response latency on a voice AI call is roughly: STT processing + model inference + TTS synthesis + audio delivery. You don't control most of that pipeline. What you do control is model inference time, and that's where prompt engineering intersects directly with call quality.

A small, fast model class like Gemini 3.x Flash-Lite sits at the low end of inference latency. It's not as capable as a frontier model on complex reasoning, but on a live call, "complex reasoning" isn't usually what's happening. The caller wants to change their delivery address, check if their package shipped, or book a table. These are information-retrieval and data-collection tasks. A fast model handles them fine, provided the prompt does the heavy lifting the model's smaller reasoning budget can't.

The trade-off is explicit: you move complexity from the model to the prompt. The model gets simpler; the prompt gets more precise.

---

## Writing terse, imperative instructions

Frontier models can infer intent from loose, principle-based instructions. Small models do better with direct commands.

Instead of: