The most common piece of feedback in a voicebot demo is some version of this: the answers are good, but the conversation feels robotic. Teams usually attribute it to the voice or to the language model.
Listening to the recordings almost always tells a different story. The model produced the right answer, the voice was fine, and the answer started about a second and a half late.
The human ear does not register that gap as poor audio quality. It registers it as "you did not understand me". This is why the first thing to design in a voice assistant is not the dialogue script but the latency budget.
How fast turn-taking actually is
There is solid evidence to anchor on here. Stivers and colleagues analysed face-to-face conversation across ten languages, including small traditional community languages and major world languages. In every language the distribution of response offsets was unimodal, peaking within 200 milliseconds of the end of the question. The variation in average gap across languages stayed inside a 250 millisecond band (Stivers et al., PNAS 2009).
People hand each other the floor inside a very narrow window. Any silence outside that window carries meaning: hesitation, disagreement, confusion, or thinking.
Telephony has its own threshold. ITU-T Recommendation G.114 states that when one-way delay stays under 150 milliseconds, interactivity is essentially transparent; 150 to 400 milliseconds is acceptable provided the impact is understood; above 400 milliseconds is unacceptable for general network planning (ITU-T G.114). That 400 millisecond figure describes transporting audio alone, before any AI component enters the picture.
A voice assistant adds to that budget. It cannot subtract from it.
Where the budget goes
Break a single turn into its parts and this is what happens between the caller falling silent and the assistant producing sound:
- Detecting that the caller has finished (VAD and endpointing)
- Transcribing the speech (ASR)
- Resolving intent and producing a response
- Querying a business system where needed (CRM, ERP, booking, order management)
- Converting text to speech (TTS)
- Carrying that audio back over the network
None of these is catastrophic on its own. The problem is that every team tunes its own component to something "acceptable" and the total lands at twice the target. If ASR takes 300 milliseconds, the model 700, the CRM lookup 400 and TTS 350, the total is 1,750 milliseconds and the caller hears an interruption.
Budget end to end, not per component. The target we work to is time-to-first-audio under one second on typical turns. Not total response time, the moment sound starts. For the person on the line, the conversation resumes when the answer begins, not when it finishes.
Endpointing is usually the most expensive part
Most teams focus on ASR and the model. The largest hidden cost is the decision that the caller has stopped speaking.
The classic approach uses a fixed silence threshold: if nobody speaks for 800 milliseconds, treat the utterance as complete. Lower it and the assistant interrupts anyone who pauses mid-sentence. Raise it and you levy a fixed tax on every single turn.
This gets worse with numbers, addresses and reference codes, which callers deliver in chunks with natural pauses between them. A fixed threshold either splits the utterance or wastes close to a second per fragment.
The fix is not to keep tuning the threshold. It is to make the endpointing decision context-aware. If the expected value is a phone number, the assistant waits because it knows how many digits are still missing. If the expected value is yes or no, 300 milliseconds is plenty. That change alone visibly reduces average turn time.
One business system call can burn the whole budget
You may have the model down to 700 milliseconds, but if the customer's CRM returns in 2.5 seconds at peak hours the conversation is already broken. That is not an AI problem; it is an integration design problem.
Three things work in practice. Fetch the customer record up front from the calling number, before the conversation starts. For genuinely slow lookups, have the assistant say that it is waiting. And put a hard timeout on every integration call, with defined behaviour when that timeout fires.
Skipping the third produces a recognisable failure: the system slows down, the assistant goes quiet, the caller says "hello?", the assistant treats that as new input, and the conversation falls apart.
Filler phrases have to be honest
Having the assistant say "let me check that for you" is a legitimate way to cover latency. Using it reflexively on every turn does not speed anything up; it just makes calls longer. Two rules:
- Play a waiting phrase only when something genuinely slow is running in the background
- Never repeat the same phrase back to back, because a repeated pattern immediately reads as an automated system
Barge-in is not negotiable
Callers must be able to interrupt the assistant. Being forced to sit through a long announcement is precisely the problem a voice assistant is supposed to solve.
Barge-in without acoustic echo cancellation is worse than no barge-in: the assistant hears its own voice, treats it as caller speech, and interrupts itself continuously. This belongs to the telephony layer and has to be solved before any dialogue design work starts.
What to measure
The most common mistake in voicebot reporting is looking at averages. A system with a 900 millisecond average turn time can still be a poor experience if ten per cent of turns exceed three seconds, and the average will never show it.
Track:
- p95 time-to-first-audio, not the mean
- False cut-off rate caused by endpointing
- Barge-in success rate
- Share of integration calls that hit their timeout
- Number of turns where the caller had to repeat information
That last one is unusually informative. Repetition is a single signal that reflects ASR quality, endpointing and dialogue design at once, and it can be computed retrospectively from call recordings.
Where latency is acceptable
Not all delay is bad. If the caller expects something to take time, a booking being created, a payment being verified, two seconds passes unnoticed. Nielsen's widely used thresholds support this: 0.1 seconds feels instantaneous, one second keeps the flow of thought uninterrupted, and ten seconds is the limit of attention (Nielsen Norman Group).
The problem is unexplained delay. A second and a half of silence right after the caller answers "how can I help you?" has no justification. The same second and a half after "let me look up your order number" goes unnoticed.
Latency budgeting is not only an optimisation exercise. It is part of dialogue design.
Before you go live
- What is the target time-to-first-audio, and is it defined on p95?
- Does the endpointing threshold vary by expected field type?
- Does every integration call have a timeout and a defined timeout behaviour?
- Have barge-in and echo cancellation been tested on real lines?
- Has transport delay on the telephony leg been measured rather than assumed?
Once these five have clear answers, most of the "it sounds robotic" feedback disappears without changing the model.
To work through the architecture against your own call volumes and integrations, start with the topology on the Voicebot Platform page.
References
- Stivers, T. et al. (2009). Universals and cultural variation in turn-taking in conversation. PNAS, 106(26). pnas.org
- ITU-T Recommendation G.114, One-way transmission time. itu.int
- Nielsen, J. Response Times: The 3 Important Limits. nngroup.com