A Real-Time Voice Application

The question and what it scores

The interviewer says: "Harbourline is moving its claims phone line to voice AI. A caller reports a car accident, the system takes the first notice of loss, and hands the case on. Design it."

They are scoring four things. Do you reason from a latency budget, in milliseconds per stage, instead of saying "it should be fast"? Can you choose between a cascaded pipeline and a speech-to-speech model and name what each costs you? Do you understand that conversation is a timing problem, with turn detection and interruption, not a text chat with a microphone? And do you know when the machine must stop and a person must take over?

The latency budget

People leave very short gaps between turns. Stivers and colleagues compared ten languages in 2009 and found that all of them minimise silence between turns, and that the average gap in each language lies within 250 ms of the cross-language mean. The figure usually quoted is a gap of about 200 ms; I did not verify that exact number in the paper, so say "a few hundred milliseconds" if challenged. A voice agent cannot match that. It can only be fast enough that the pause reads as thinking and not as a dropped call.

Production targets are looser than the human gap. Twilio's published guide for cascaded agents, as of November 2025, sets a mouth-to-ear turn gap of 1,115 ms as the target for a launch and 1,400 ms as the limit. Mouth-to-ear means from the moment the caller stops speaking to the moment they hear the reply, network included. You may hear 500 to 800 ms quoted in interviews; I found no primary source for it, so treat it as a stretch goal that usually needs speech-to-speech models, not as the bar for a first launch.

The same guide gives per-stage figures. The three model stages take 825 ms at target and the rest of the 1,115 ms is transport and buffering:

The useful move is to run the arithmetic live. This small program does it:

TARGET = {"stt": 350, "llm_ttft": 375, "tts_ttfb": 100}
LIMIT = {"stt": 500, "llm_ttft": 750, "tts_ttfb": 250}
MOUTH_TO_EAR_TARGET, MOUTH_TO_EAR_LIMIT = 1115, 1400


def turn_gap(stages: dict[str, int], transport_ms: int, tool_ms: int = 0) -> int:
    return sum(stages.values()) + transport_ms + tool_ms


transport = MOUTH_TO_EAR_TARGET - sum(TARGET.values())
print("transport and buffering:", transport, "ms")                       # 290
print("all stages at limit:", turn_gap(LIMIT, transport), "ms")            # 1790
slow_llm = {**TARGET, "llm_ttft": LIMIT["llm_ttft"]}
print("LLM alone at limit:", turn_gap(slow_llm, transport), "ms")         # 1490, over 1,400
print("on target + 800 ms tool call:", turn_gap(TARGET, transport, 800))  # 1915

Two lessons come out. One slow stage breaks the whole budget: the model alone at its upper limit puts the turn over 1,400 ms. And a single 800 ms tool call sends an on-target turn to nearly two seconds, which is why tool calls need their own strategy, below. Measure the tail, not the average. A caller remembers the one 3-second silence, so track the 95th percentile turn gap per stage. Chapter 10 covers the model-side levers: streaming, caching and routing.

Cascaded or speech-to-speech

A cascaded pipeline streams caller audio to a speech-to-text service, passes text to a language model, and streams the reply tokens to a text-to-speech service. You own every seam. A speech-to-speech API takes audio in and returns audio out from one model, and the session handles turns, tools and interruptions for you. OpenAI's Realtime API is an example, offered over WebRTC for browsers and WebSocket for servers; its guide says to choose the chained path when each stage must be inspectable or replaceable, and otherwise to use the single-model session.

For Harbourline I recommend cascaded, and the reason is not latency. A first notice of loss is a regulated record. The company needs the exact words the caller said, a text-level guardrail on every reply, and the ability to run the claims logic with its own model and prompts. Those are control and audit properties, and cascaded gives them by construction. It flips when the product is a conversational companion where feel matters more than record, or when a speech-to-speech model meets your latency target and your compliance team accepts a vendor-transcribed record. A hybrid is common: speech-to-speech for the conversation, with a parallel transcription stream saved as the record.

Turn-taking: endpointing, barge-in and the cost of getting it wrong

Before the model can answer, something must decide the caller has finished. That decision is endpointing, and it is the single biggest controllable part of the gap. A fixed silence threshold has a sharp trade-off. Set it short and the agent cuts in when the caller pauses to recall a plate number; set it long and every turn drags. Deepgram's endpointing parameter is in milliseconds; its docs, as of November 2026, say the default of 10 ms suits short chatbot utterances and recommend 300 to 500 ms for callers who pause mid-thought. Reading a policy number aloud is exactly that case.

Better systems stop relying on silence alone. Semantic turn detection looks at what was said, so "my policy number is" waits and "yes" does not. Another useful trick is to start the model on the interim transcript and discard the work if the caller keeps talking. That trades compute for latency. Streaming speech-to-text also returns two flags that are easy to confuse: is_final marks a finalised segment, and speech_final marks a detected pause. A long utterance produces several final segments before the pause, so concatenate segments until the pause arrives instead of acting on the first final one.

Barge-in is the caller talking over the agent. Handling it takes four steps and candidates usually name only the first.

1. Detect speech while the agent is speaking, and require a minimum duration, perhaps 200 to 300 ms of voiced audio, so a cough or the echo of the agent's own voice does not cut it off. Tune on real calls; the right value depends on the line.
2. Stop playback at once. On a Twilio Media Stream, the clear message empties audio Twilio has buffered; without it, queued audio plays to the end.
3. Cancel the model and speech generation in flight, or you keep paying for a reply nobody hears.
4. Correct the conversation history to what the caller actually heard. If the agent produced three sentences and the caller interrupted during the second, the history should hold one and a half, not three, or the model will assume the caller heard a deductible figure they did not. Twilio's mark messages make this possible: send one after each sentence of audio, and the marks that return before the clear are the sentences that played.

# Sketch, not run here: barge-in on a Twilio bidirectional media stream (asyncio + a websocket).
async def on_caller_speech_start(ws, state):
    if not state.agent_speaking:
        return
    await ws.send_json({"event": "clear", "streamSid": state.stream_sid})   # stop buffered audio
    state.tts_task.cancel(); state.llm_task.cancel()                        # stop paying for it
    # state.heard holds marks echoed back before the clear: those sentences were played in full
    state.history[-1] = " ".join(state.sentences[: len(state.heard)])      # keep only what was heard
    state.agent_speaking = False

Backchannels such as "mm-hm" are the hard case. A caller saying "yes" while the agent reads a figure is agreeing, not interrupting. Most stacks do not treat short acknowledgements as barge-in at all; say that you would test this with recorded calls.

Tools mid-conversation, and filling the silence

The first notice of loss needs tools: look up the policy, check it was in force on the date of loss, create the claim. A tool call costs hundreds of milliseconds to seconds, and silence on a phone is worse than in chat. The budget above shows an 800 ms call pushing the turn to nearly two seconds.

Three techniques, in order of preference. First, make the call earlier: look up the policy as soon as the caller gives a name or number, while they are still describing the accident, so the answer is waiting. Second, speak a short filler that is true, such as "Let me pull up your policy", the moment a slow tool starts, and make sure the filler is cancelled by barge-in like any other speech. Third, keep tools that run over a second off the live path, and say "I will have someone confirm that" instead. Do not use a filler for every turn; callers notice the pattern.

Voice changes tool design in two further ways. Tool results must be written for the ear: read back a policy number in groups, never as a 20-character string, and confirm digits and dates before writing them to the claim. Speech-to-text will mishear them. For long identifiers, offer keypad entry; Twilio streams report keypad presses as separate dtmf messages on bidirectional streams. The same loop and permissions design applies as in text; see the customer-service agent and agent or workflow. The phone agent should gather facts and create the claim. It should not decide coverage; that stays with the claims system and an adjuster, as in the claims agent chapter.

Getting the call to your code

Two transports matter. A phone caller arrives over the public telephone network, and the provider hands you the audio. Twilio Media Streams opens a WebSocket to your server and sends base64 audio as 8,000 Hz, single-channel, mu-law, with connected, start, media and stop events. That detail matters: telephone audio is narrowband, so a speech-to-text model trained on 16 kHz or wideband audio will do worse on it, and the text-to-speech output has to be encoded back to mu-law at the same rate. Test your recogniser on 8 kHz recordings of real accents and car noise, not studio audio.

For app and web users, WebRTC gives wideband audio, built-in echo cancellation and lower latency, and is what browser-side speech-to-speech APIs use. A SIP trunk into your own media server is the third route, when Harbourline has an existing contact-centre platform and wants calls to arrive through it. For the phone line, I would start with a provider's media stream: it removes the telephony work, and the cost is a WebSocket hop and the provider's per-minute price. Moving to SIP into your own stack pays off at scale, or when you need control over where the call is routed and recorded.

Handoff to a human, recording and consent

Decide the handoff triggers before the prompt. Injury, a fire, a child in the car or a caller in distress goes to emergency guidance and a human immediately, ahead of any claim script. Other triggers are measurable: the caller asks for a person, the agent fails to capture a field twice, speech-to-text confidence stays low for several turns, or the caller sounds angry. The handoff must carry context, so the adjuster sees the transcript and the fields gathered, and the caller does not repeat the story. Always keep a path to a person; a voice agent with no exit is a complaint generator.

Recording is a legal question as well as an engineering one. Some jurisdictions require every party's consent to record a call and others only one; Harbourline operates in the US and the UK, so the rules differ by state and country. Practice is to say at the start of the call that it is recorded and that an automated assistant is on the line, and to take legal advice on the wording. Then treat the audio as sensitive personal data: separate retention for the audio and the transcript, redaction of card and bank details before they reach logs, and access limited to the claims and quality teams. Speaking the disclosure also costs seconds of every call, so keep it short.

A strong answer, in order

1. Set the constraint: callers expect an answer within about a second and a half at most, so the budget is 1,115 ms mouth to ear with a 1,400 ms ceiling, tracked at the 95th percentile.
2. Pick cascaded, because the first notice of loss is a record that needs exact text, text-level guardrails and the company's own claims logic. Say when you would choose speech-to-speech.
3. Stream every stage. Endpoint at about 400 ms with semantic help, because callers read out numbers slowly.
4. Handle barge-in in four steps: detect, clear, cancel, correct the history.
5. Prefetch the policy early, use short true fillers for slow tools, and never decide coverage on the call.
6. Carry the call over a provider media stream, tune the recogniser for 8 kHz audio, and escalate on injury, repeated failure or request, with context.
7. Disclose recording, retain audio and transcript separately, and evaluate on recorded calls before launch.

Interview probes

✓ Check yourself

Answer from memory before opening each one.

1. Speech-to-text, the model and text-to-speech each hit their upper limits on the same turn. Without computing the sum, will the turn stay within the 1,400 ms ceiling, and what does that say about how to set targets?

   Show answer

   No. The upper limits sum to 1,500 ms before any transport, and with the 290 ms of transport the turn reaches about 1,790 ms. Even the model alone at its limit breaks the ceiling. So the limits cannot be treated as independent budgets; track the tail per stage and the total. See the latency budget.

2. A caller reads a 12-digit policy number in groups of three, pausing between groups. The agent keeps cutting in after each group. What setting is wrong, what does the fix cost, and what is a better fix than a larger number?

   Show answer

   The endpointing silence threshold is shorter than the caller's pauses. Raising it delays every turn by the same amount. A better fix is turn detection that looks at content, so an incomplete number waits and a complete answer does not, or keypad entry for the number. See Turn-taking.

3. After a barge-in the agent says "As I mentioned, your deductible is 500", but it was interrupted before saying the figure. Which step in the barge-in handling was skipped, and why did stopping the audio not prevent this?

   Show answer

   The history was not corrected to what the caller heard. Stopping playback only halts audio; the model's record still holds the full planned reply. Marks sent after each sentence tell you which sentences played, and the history should be trimmed to those. See Turn-taking.

4. Your recogniser scores well on your test set but mishears plate numbers on live calls. Name two likely causes that have nothing to do with the model.

   Show answer

   Phone audio arrives as 8 kHz mu-law, narrower than the audio the test set used, and callers speak in cars, with noise and accents the set lacks. A third is acting on the first final segment of a long utterance. Test on recorded phone calls. See Getting the call to your code.

5. A product manager wants the lowest possible latency and proposes a speech-to-speech model for the claims line. What do you ask, and what would make you agree?

   Show answer

   Ask where the exact record of the call comes from, whether every reply can be checked by a guardrail before it is spoken, and whether the claims logic can use the company's own model. Agree if a parallel transcript serves as the record, the guardrail needs can be met, and the cascaded stack cannot reach the latency target. See Cascaded or speech-to-speech.

Takeaway

• Voice is designed from a per-stage latency budget; Twilio's published cascaded target is about 1.1 seconds mouth to ear, and one slow stage breaks it.
• Cascaded pipelines give exact text, per-stage control and auditability; speech-to-speech gives a lower latency floor and better prosody at the cost of control.
• Endpointing is the largest controllable delay, and a fixed silence threshold always trades interruptions against lag.
• Barge-in means detect, stop audio, cancel generation and correct the history to what the caller heard.
• Slow tools are started early, covered by a short true filler, or moved off the live path; digits are confirmed or entered by keypad.
• Telephone audio is 8 kHz mu-law, so recognisers must be tested on real phone recordings.
• Safety, handoff with context and recording consent are designed first, and the phone agent gathers facts but does not decide coverage.

A voice agent is one case of putting a probabilistic model behind a promise other systems rely on, which is the subject of the next chapter. → An LLM Behind a Reliable Contract

Further reading

• A Guide to Core Latency in AI Voice Agents (Cascaded Edition) — Twilio, 2025. The per-stage targets and limits used in this chapter.
• Twilio Media Streams WebSocket messages — the audio format, mark, clear and dtmf messages. Good for barge-in on telephony.
• Deepgram: endpointing and interim results — endpointing in milliseconds, is_final versus speech_final.
• OpenAI voice agents guide — a vendor's own comparison of speech-to-speech and chained designs.
• Universals and cultural variation in turn-taking in conversation — Stivers et al., PNAS 2009. The cross-language evidence for minimised gaps between turns.