Prompting Architecture For Voice Agents

# Prompting architecture for production voice agents

I've shipped a dozen or so production voice agents on ElevenLabs Conversational AI. Before the first one went live, I assumed writing a voice prompt was roughly like writing a chat prompt, maybe a little shorter. That assumption caused bugs in about the first three calls we listened to. Voice is a fundamentally different medium, and the prompt architecture has to account for that from the start.

---

## Why voice changes everything about prompt structure

In a chat interface, a user can scroll back, re-read the bot's last message, and take their time. On a phone call, the interaction is ephemeral and linear. The caller heard what they heard, and they're already formulating a response before you're done speaking. If the caller interrupts (and they will), the agent has to handle partial context and move on.

Latency matters too. Every instruction in your prompt costs tokens, and on a live call, token processing happens on every turn. Bloated prompts directly translate to longer response latency, which translates to awkward silences, which translates to callers saying "hello?" or hanging up. Structure your prompt for the machine's reading speed.

---

## The anatomy of a production system prompt

ElevenLabs defines a canonical six-section structure for agent system prompts. Each section has a specific heading and purpose:

**`# Personality`.** Who the agent is and what role it plays. Keep this to two or three sentences. This is the agent's identity: name, company, purpose. No backstory.