When choosing a transcription API, accuracy is usually the first metric. The problem is that providers measure accuracy on different data, so headline numbers do not line up.

The open baseline: Whisper

Whisper (Radford et al., 2022) is an open speech recognition model trained on a large, diverse dataset. On the standard LibriSpeech benchmark, the large model achieves a word error rate (WER) of 2.7% on test-clean and 5.2% on test-other.

Those numbers are reproducible and well-documented, which makes Whisper a useful reference point. It is also free to run yourself, subject to compute cost.

Cloud APIs

Cloud providers (Google, AWS, Azure, and others) publish their own accuracy figures, but typically on their own evaluation sets and with their own post-processing. That means a provider’s “WER” and Whisper’s WER are not measuring the same thing.

Beyond accuracy, cloud APIs differ on features:

CapabilityWhisper (self-hosted)Typical cloud API
Word error ratePublished on public benchmarksPublished on provider datasets
Speaker diarizationNot built inOften included
Streaming / real-timeRequires custom setupUsually supported
Language coverageBroad, fixed modelOften broader, updated frequently
Operational costYour computePer-minute pricing

How to choose

Transcription choice factors Whisper (open) transparent WER baseline Cloud APIs features + managed ops Benchmark on your own audio before committing to a provider

VERDICT

Whisper is the honest, reproducible accuracy baseline. Cloud APIs win on features and operations. The only accuracy number that matters is the one measured on your own audio.

Sources