Test, Measure and Fix a Live Voice Agent

Twilio will tag a call high_latency when round-trip time exceeds 400 ms. LiveKit publishes a three-term formula for a voice agent's response delay. Between those two numbers sits a gap that nobody's documentation covers, and that gap is most of what a caller actually experiences.

This part is about what you can measure with published definitions, what you can only test, and where the vendor docs go quiet.

The only response-time formula anyone writes down

LiveKit's metrics documentation is the one place in this stack that defines a latency budget in named fields. Four of them matter:

• end_of_utterance_delay: "Time (in seconds) from the end of speech (as detected by VAD) to the point when the user's turn is considered complete."
• transcription_delay: "Time (in seconds) between the end of speech and when the final transcript is available."
• ttft: "The amount of time (seconds) that it took for the LLM to generate the first token of the completion."
• ttfb: "The amount of time (seconds) that it took for the TTS model to generate the first byte of its audio output."

The sum is written out in the docs as a single line:

total_latency = eou.end_of_utterance_delay + llm.ttft + tts.ttfb

Read the verb attached to it. The page describes this as a way to "approximate total latency by summing individual pipeline metrics." It is an approximation, and its boundaries are worth stating out loud, because the formula starts when your server decides the caller stopped speaking and ends when the first byte of audio is generated. Network transit, telephony transcoding and playback buffering are all outside it. A pipeline that looks healthy on these three numbers can still feel slow on a phone.

The correlation key is the useful operational detail. Metrics from different stages carry a speech_id, which is what lets you attribute a slow turn to the LLM rather than the TTS rather than the turn detector. Without it you have three unrelated averages and no way to say which one broke.

Twilio prints thresholds for the half you do not control

The other half of the picture comes from the carrier, and Twilio's call tags reference is unusually specific about where the lines are drawn. At the gateway edge:

At the SDK edge the tags differ: high_latency becomes "Round Trip Time (RTT) > 400 ms", and low_mos fires at "Mean Opinion Score (MOS) < 3.5".

Two of these are voice-agent specific in a way the docs do not spell out. silence on an agent call usually means the socket to the model dropped and the caller is listening to nothing, which is the single worst failure this system has. And pstn_short_duration, calls completing under ten seconds, is the closest thing anyone publishes to a measurement of your greeting. A rising count there is callers hearing the first sentence and hanging up. Nothing in the model vendors' documentation gives you that signal.

Three test shapes that run without a human on the line

ElevenLabs' agent testing documents three categories, and they answer different questions:

• Simulation testing "Runs end-to-end, multi-turn conversations with a simulated user." You "Define the outcome that should count as a pass."
• Next Reply (Scenario) testing "Validates the agent's next response against success criteria," set by describing what the response should achieve including "expected behavior, tone, and actions."
• Tool Call testing "Ensures the agent calls the right tool with the right parameters."

Tests run three ways: from the Tests tab with Run All Tests or individually, from the CLI with elevenlabs agents test [agent-id], or through the Create test and Run tests API endpoints.

The feature that changes how you read results is probabilistic testing, which runs "the same test multiple times and reporting a pass rate," with results bucketed by failure reason rather than shown as individual transcripts. That framing is the correct one for a voice agent. A single passing run tells you almost nothing about a model that samples its own output. Put the tool-call tests in CI and watch the pass rate move, not the last transcript.

Interruption quality is five parameters, not a score

No vendor here publishes an interruption metric. What LiveKit gives you instead is the set of dials, and tuning them is the measurement:

• min_duration, the minimum speech duration in seconds that registers as an interruption.
• min_words, the "Minimum number of words to be considered as an interruption. Only used if STT is enabled."
• false_interruption_timeout, the silence duration before an agent_false_interruption event fires.
• resume_false_interruption, "Whether to resume the agent's speech after a false interruption is detected."
• discard_audio_if_uninterruptible, which drops buffered audio when the agent is speaking and cannot be interrupted.

One trap sits in the units. The docs specify false_interruption_timeout in seconds for Python and milliseconds for Node.js, and the same split applies to max_duration on user turn limits. A value copied between a Python prototype and a Node service is wrong by a factor of a thousand in one direction or the other, and the symptom is a barge-in setting that appears to do nothing.

The agent_false_interruption event is the closest thing to a real interruption metric on offer. Count it per call. A high rate means your min_duration or min_words threshold is treating background noise as a caller trying to speak.

A handoff needs a trigger, a destination and a type

Every serious agent needs a documented way to stop being an agent. ElevenLabs' transfer_to_number system tool carries a condition field alongside transfer_destination and transfer_type, so the trigger is configuration rather than a sentence buried in the prompt. On the raw API side, Grok exposes two REST calls against a live SIP call, POST /v1/realtime/calls/$CALL_ID/refer with a target_uri, and POST /v1/realtime/calls/$CALL_ID/hangup.

The failure mode you are more likely to hit first is a dropped WebSocket rather than a transfer. xAI's speech-to-speech docs handle that with session resumption: opt in with resumption.enabled: true, store the id from conversation.created, and reconnect with ?conversation_id=<id>. Two constraints are published. Both sides must opt in, since "No history replays unless the resuming session also sends resumption.enabled: true." And the cache expires: "History is dropped after 30 minutes of inactivity."

The same page documents an audio bug you will hear before you can name it. When the model calls a tool, the server sends all audio deltas first, then the function call events alongside response.done. Send your function_call_output and response.create immediately and the next response starts while the previous audio is still playing. The documented fix is to wait until playback of the current turn is complete before sending response.create, and to show a thinking indicator meanwhile.

Where the documentation stops

Three gaps are worth naming, because reading around them wastes days.

xAI publishes no latency figure and no per-turn timing events. Its speech-to-speech page has a Best Practices section, and every entry is advisory: start the socket "as early as possible," "Monitor WebSocket health and use exponential backoff if needed," "Match input/output format (24 kHz PCM) to avoid resampling." There is no metrics event to subscribe to, no published time-to-first-audio, and nothing resembling a latency commitment on the pricing page either.

ElevenLabs' conversation analysis page names five capabilities, success evaluation, data collection, sentiment analysis, coaching and conversation search, and describes sentiment as "positive, neutral, or negative." It does not publish the result vocabulary for success evaluation on that page, so what a failed evaluation looks like in an export is something you find out by exporting one.

And no vendor in this stack spans the whole path. LiveKit's formula stops at the first generated byte. Twilio's tags start at the carrier edge. The middle, your server to the caller's handset, is measured by neither, and there is no published document that joins them. Instrument both ends, log a call identifier that appears in both systems, and accept that the number in the middle is inferred rather than reported.