WER vs DER vs SA-WER: the three speech metrics, and why two of them hide the failure that matters

We spent months looking at the wrong number. Our word error rate was solid, transcripts read clean, and speaker attribution was correct on 62.5% of words. One in three words was in the wrong person's mouth, and nothing we were measuring said so.

The reason is that the two metrics everyone quotes measure different things, and the failure lives in the gap between them.

Word error rate measures words and ignores speakers

WER is substitutions plus deletions plus insertions, over the number of words in the reference. It is the number every vendor publishes, and it is genuinely useful — for transcription.

It is also completely blind to who was speaking. Transcribe an entire call perfectly, attribute every word to the wrong participant, and WER does not move. The metric has no opinion about speakers because speakers are not in its definition.

Diarization error rate measures time and ignores words

DER is the fraction of audio time that is misattributed: speech the system missed, silence it labelled as speech, and time it gave to the wrong speaker. It is the standard diarization metric and it is where most people stop.

Here is the trap: DER is weighted by duration. Long turns dominate the score. But in a real conversation the meaning lives in the short ones.

• “No, that's not what I ordered.”
• “How much is it?”
• “Yeah, go ahead.”

A monologue that runs ninety seconds and a rejection that takes one both count in seconds, so the rejection is worth roughly one percent of the monologue. You can post an excellent DER and still miss every moment a downstream analytics product was built to find.

A 5% DER is not 5% of the meaning. It is 5% of the seconds, and the seconds that carry meaning are the shortest ones.

The metric that actually predicts usefulness

What a diarized transcript is for is answering who said what. The metric family that measures that directly is speaker-attributed WER — SA-WER, and its close relative cpWER (concatenated minimum-permutation WER, from the CHiME challenges). The idea is simple: compute error per speaker after aligning speakers, so a correctly transcribed word attributed to the wrong person counts as an error, exactly as a user would perceive it.

Almost nobody publishes it. Ask your vendor for WER and you will get it. Ask for DER and you might. Ask for SA-WER and the conversation usually changes subject.

What we found when we measured it

Our diarization assigned speakers by segment overlap: take a transcript segment, see which diarization span covers most of it, give the whole segment to that speaker. It is the obvious approach and it works fine while people take turns politely.

People do not take turns politely. The moment two speakers overlap — which on real calls is constant — the entire segment goes to one of them, and every word of the other speaker's interjection is misattributed.

Two changes fixed it.

• Resolve at the word level.Every word carries its own timestamp, so attribute each word independently against the diarization spans instead of inheriting the segment's label.
• Delete the smoothing pass.We had a step that “corrected” short A-B-A interruptions on the assumption they were clustering noise. They were not noise. That is how people talk, and the smoothing was actively destroying accuracy on exactly the turns that carry the most meaning.

Measured on real client audio with manual ground truth, attribution accuracy went from 62.5% to 96.9%.

Word error rate did not change. It had never been wrong. We had simply been reading a metric that could not see the problem, and feeling good about it.

What to ask a vendor

If you are buying speech-to-text for anything with two people in it — call centre QA, sales intelligence, medical scribing, legal hearings — WER alone will not tell you whether the product will work. Three questions worth asking:

• Is speaker attribution resolved per word, or per segment? Segment assignment is cheap and fails on overlap.
• What is your DER, and is it measured on audio with realistic overlap? A DER measured on clean turn-taking predicts nothing about a live call.
• Do you measure anything in the SA-WER family? If not, nobody has checked whether the right words land on the right speaker.

If you are recording the calls yourself, there is a cheaper fix than any of this: keep both channels. A stereo recording with the agent on one side and the customer on the other gives exact attribution by construction, with nothing to infer. A mono mixdown throws away separation you already had, and no model recovers two voices that have already been summed.