WER, MOS, Latency: The Voice Agent Metrics That Miss Emotion

Here is an uncomfortable claim about voice agent evals: the metrics almost everyone ships with can be maxed out by a system that never listens like a human. Transcribe the audio, reason on the text, speak fluent output. Word error rate stays under 5%, mean opinion score clears 4.5, task completion looks great, and time to first token lands under 500ms. Every box is green. And the agent has quietly ignored the thing that made the input speech instead of a chat message.

That is the gap RW-Voice-EQ Bench put a spotlight on. Its finding that many speech-to-speech agents remain transcript-driven, with audio access but no use of vocal affect, is the most important result in voice AI evaluation right now, and the commercial tooling has barely engaged with it.

The scorecard measures output, not perception

Read the dominant voice agent metrics guides side by side. Hamming AI, ElevenLabs, Cekura, Langfuse, Braintrust, Maxim AI. They largely agree, and they are largely right about the plumbing: you need end-to-end tracing across ASR, NLU, LLM, and TTS, because failure modes differ per layer and STT errors cascade into wrong intents and failed tasks. The Daily.co STT benchmark makes that cascade concrete.

But look at what these frameworks actually score on the human side. MOS measures how natural the agent sounds. Sentiment and CSAT measure outcomes after the fact. Nowhere is there a metric that asks the direct question: did the agent perceive the emotion, prosody, and paralinguistic cues in the user’s speech, and did it change its behavior because of them?

MOS naturalness of output is not affect-use. A text-to-speech engine can sound warm while the reasoning core treats a panicked caller and a bored caller identically. We are measuring how good the mouth sounds and skipping whether the ears worked.

Why this matters most where the money is

These tools are being sold hardest into healthcare, customer support, and sales. Those are exactly the domains where the input carries load-bearing emotion. A patient who says “I’m fine” with a shaking voice is not fine. A customer whose pace and pitch spike is about to churn. An agent that lands the words and misses the affect passes the eval and fails the human.

That is the systematic error. When affect-use is not scored, the eval stack rewards transcript-driven agents over affect-aware ones, because the affect-aware agent pays a latency and complexity cost for a capability no metric credits.

A formula-backed way to measure affect-use

Here is a concrete proposal the sources do not offer. Treat affect-use as a controlled sensitivity test, not a vibe.

Take a single transcript. Render it in multiple prosodic conditions: neutral, distressed, angry, rushed. Same words, different vocal affect. Feed each version to the agent. Then measure the divergence in the agent’s response across conditions, holding the text constant.

Affect-use score = behavioral divergence across affective conditions, given identical transcripts.

If the agent produces near-identical responses regardless of how the words were spoken, its affect-use score is near zero. It is a chatbot with a microphone. If it de-escalates for the angry render, slows down and reassures for the distressed one, and gets efficient for the rushed one, the score climbs. This is reproducible, it isolates the exact capability RW-Voice-EQ flags, and it can run inside the same simulation harness Coval and Cekura already advocate for scenario testing.

The elegant part: it doubles as a diagnostic. Run the same test with audio stripped to text only. If scores do not change, you have proof your pipeline is transcript-driven by construction.

The architecture and cost angles nobody prices

Two more things the metric taxonomies leave black-boxed.

First, latency is reported as a number, never attributed to a tunable component. i-LAVA showed that TTS, specifically the number of RVQ iterations in the decoder, dominates real time factor. So when your P99 latency spikes, the fix may live in the vocoder, not the LLM. Evals should attribute latency to the stage that owns it, or teams optimize blind.

Second, the industry pitch of running thousands of simulated conversations has an unpriced bill. Every simulated call plus every LLM-as-a-judge pass is compute and tokens. If your judge model costs a few cents per scored turn and you run multi-turn regression suites nightly, cost per eval becomes a real line item, and it caps how often you can afford to test. Nobody publishes that number, and it shapes what “continuous production voice AI monitoring” actually means in practice.

One more unvalidated assumption

Every framework leans on LLM-as-a-judge for subjective dimensions like naturalness and empathy. None of the practitioner sources validate that judge against human raters on the same calls. For affect specifically, that is a problem: a text-based judge reading a transcript is structurally blind to the very signal we are trying to score. If you judge affect-use, your judge needs the audio too.

The takeaway

A voice agent testing framework that tracks WER, MOS, task completion rate, turn detection and barge-in, agent talk ratio, and latency is necessary and not sufficient. Until affect-use is a first-class, formula-backed metric, conversational AI quality scores will keep certifying agents that hear words and miss people.

The fix is not exotic. Hold the transcript constant, vary the vocal affect, measure the divergence. Add it to your suite next sprint.

So here is my question: if you stripped the audio from your voice agent and ran it on transcripts alone, how much would your eval scores actually drop? If the honest answer is “barely,” what exactly are you selling?