Latency budgets for voice agents
Users forgive a slow page load. They do not forgive a slow reply in a conversation. Here is how to spend a 500 ms budget without wasting it on the wrong stage.
In a chat interface, a two-second wait is annoying. In a voice interface, it reads as a dropped connection. The difference is that speech has no visible progress indicator: silence is the loading state, and silence is indistinguishable from failure.
A reasonable target for a conversational turn is under 500 milliseconds from end of user speech to first audio byte. That budget has to cover transcription, model inference, and synthesis, and it is almost always spent in the wrong place.
Where the time goes
Transcription is usually the cheapest stage, especially for short utterances. Text generation is the least predictable, because it depends on the model and on how many tokens you ask for. Synthesis is the stage teams most often over-provision.
- Endpointing: 150–300 ms, and the biggest source of perceived lag.
- Transcription: 50–150 ms for a short clip on a warm connection.
- Generation: highly variable; stream tokens instead of waiting for a full response.
- Synthesis: 100–250 ms to first byte for a short first sentence.
Stream everything
The single biggest win is starting synthesis before the full response exists. Split the model output at the first sentence boundary and send that fragment to the speech endpoint while the rest is still generating.
This changes the failure mode too: if the user interrupts, you have only synthesized a sentence or two, not a paragraph you will throw away.
Measure the turn, not the endpoint
Per-endpoint latency numbers are easy to produce and easy to mislead with. The number that matters is the wall-clock gap between the end of user speech and the first audible byte, measured on a real connection from the region your users are in.
Log that number on every turn. Averages hide the tail, and the tail is what users remember.
Try it with your own text.
322 voices across 75 languages, plus transcription, on one API key. Free tier included.
Start free