Speaker Diarization Explained: How Machines Label Who Is Talking

Speaker diarization figures out who said what in a recording. At Layer3Labs, we build call and intake automation for other businesses, and diarization often determines whether a workflow can do anything useful with the audio. Transcription gives you the words; diarization separates them into turns and assigns each turn a label, Speaker 1, Speaker 2, and so on.

The words are usually accurate. The speaker labels may arrive late, be wrong, or appear only after the recording ends. That gap shapes what your application can do while the conversation is still happening.

Major speech-to-text application programming interfaces (APIs) price speaker labels in three ways. Some include them with the transcription model, some charge for them as a per-hour add-on, and Whisper does not offer them at all.

What Speaker Diarization Is

Diarization answers one question: who spoke when. The system listens for changes in voice characteristics, cuts the audio at those change points, groups the segments that sound like the same person, and attaches a label to each group.

It does not know names. A diarization system returns Speaker 1, Speaker 2, Speaker 3, and your application maps those to real people. Some meeting tools do that mapping against the calendar invite, and others ask each person to introduce themselves.

It is also a separate job from transcription, which is why so many stacks run two models. Whisper writes accurate text and has no speaker labelling at all. Teams building on it pair it with an open diarization library such as pyannote and stitch the two outputs together by timestamp.

That stitching is the part people underestimate. You are aligning two models that disagree about where words start and stop. Every disagreement shows up as a word attributed to the wrong person at a turn boundary.

• Transcription answers what was said. Diarization answers who said it.
• Output is anonymous labels like Speaker 1, so your application maps them to people.
• Some models do both jobs in one pass. Others need a second model and a timestamp merge.
• The merge at turn boundaries produces most attribution errors, so budget review time for it.

How Diarization Accuracy Is Measured

Diarization accuracy is reported as diarization error rate (DER), the share of audio time attributed to the wrong speaker. A 10% DER means one minute in ten carries a wrong label. Lower is better. The number is not comparable to word error rate, which counts words rather than seconds.

DER bundles three different mistakes into one figure. The system can miss speech entirely, it can label silence or noise as speech, or it can put the right speech under the wrong speaker. Two models with the same DER can fail in very different ways, so check which of the three a vendor is reporting before you compare two figures.

The benchmarks you will see cited are AMI and VoxConverse. AMI is recorded meetings, and it comes in two flavours that matter. AMI-IHM uses individual headset microphones. AMI-SDM uses a single distant microphone in the room, and it is much harder, because overlapping voices arrive mixed rather than on separate channels.

For a current reference point, Meta reports a 17.5% average DER for Muse Voice Transcribe across AMI-IHM, AMI-SDM and VoxConverse. Numbers in that range are normal for hard multi-speaker audio, and they carry a practical instruction: plan for a person to correct labels on anything that carries legal or clinical weight.

Streaming Diarization vs Batch Diarization

Labels arriving during the call, rather than after it, changes your architecture. Batch diarization waits for the complete recording, which lets it compare every segment against every other one and cluster them with full information. Streaming diarization has to decide who is talking now, using only the audio that has already played.

Batch is more accurate for the same model, because it can revise an early guess once it has heard more. It is also useless for anything live. A support agent cannot be prompted mid-call by a system that will not know who the caller was until the call ends.

Streaming diarization used to mean running a second real-time model alongside transcription and paying for both. Muse Voice Transcribe changed that shape by attributing speech inside the recognition model, which removes the merge step and the second bill together.

Pick batch when your product analyses finished recordings: call-quality scoring, meeting summaries sent afterwards, research interviews. Pick streaming when something has to happen while people are still talking, such as live captions, agent prompts, or a voice agent that needs to know whether the customer or the representative is speaking.

Which Speech-to-Text APIs Include Speaker Diarization

The spread between these three pricing shapes is wide enough to change which vendor is cheapest for your workload. At 500 hours a month it separates a $90 bill from a $285 one.

Read those rates as a shape rather than a quote. Every vendor listed has changed prices in the past year, and each publishes current figures on its own pricing page, linked in the table.

One billing detail catches people out. AssemblyAI charges streaming by how long the connection stays open rather than by how much audio you send, so an idle websocket still costs money. In a support queue, open the socket on first audio rather than when the session starts, or you pay for hold music.

Where Speaker Diarization Fails

Diarization degrades in predictable ways, and knowing them saves a week of blaming the wrong component. Audio conditions matter more than model choice in most real deployments.

Overlapping speech is the hardest case. When two people talk at once, the audio carries both voices in one signal, and most systems assign the segment to whoever is louder. Meetings where people interrupt each other produce far worse labels than meetings where they take turns.

Phone audio makes the similar-voices problem worse. The narrow frequency band strips out much of what distinguishes one voice from another, so two speakers with close pitch and accent merge into a single label. The transcript then reads as one person talking to themselves.

In the client intake workflows we have automated for law firms, the practical fix was never a better model. It was recording each participant on a separate channel wherever the phone system allowed it, which turns diarization from a guess into a lookup.

• Overlapping speech: the louder voice wins the segment, so interrupt-heavy meetings label badly.
• Similar voices: two speakers merge into one label, and phone-band audio makes it more likely.
• Short turns: a one-word answer gives too little signal, so the yes and no replies that carry the decision land on the wrong speaker.
• Unknown speaker count: systems that must guess how many people are present drift as the count grows.
• Separate channels beat every software fix, when your recording setup can give you them.

How to Choose a Diarization Approach

Start from what your product does with the labels. The benchmark table comes second. Three questions settle most decisions, and they are listed at the end of this section.

Two people on separate channels barely need diarization at all. Twelve people around one conference microphone need the best model you can get, plus a human review step on top.

Who this is not for: if you transcribe single-speaker audio, such as dictation, voice notes, or a solo podcast, diarization adds cost and a failure mode and buys you nothing. Turn it off and spend the budget on transcription accuracy instead.

What would change this advice: a model that handles overlapping speech cleanly, or bundled streaming diarization becoming standard across every major API. Both look likely within a year, and either one removes the cost calculation above. To get the diarization question answered against your own call recordings rather than a benchmark, book a workflow audit.

• Does anything need to react while people are talking? If yes, you need streaming diarization and the field narrows to the models that do it live.
• Can your audio leave your own infrastructure? If it cannot, you are self-hosting Whisper with pyannote and paying in hardware and engineering hours.
• How many speakers, and how clean is the recording? The answer sets how much review time you have to budget on top of the model.