The problem
Whisper transcribes speech impressively well right out of the box — no training required. The more useful question for production is which size to actually deploy, and whether a bigger model is worth its cost. Three Whisper sizes were evaluated on 100 LibriSpeech clips split across an easy and a hard audio condition.
Key results
Three models, two conditions
Whisper tiny (37.2M parameters), base (71.8M), and small (240.6M) were each run on 50 clips from LibriSpeech's test-clean (easier audio) and test-other (harder audio) splits, measuring both Word Error Rate and total inference time.
| Model | WER (clean) | WER (other) | Time (clean) | Time (other) |
|---|---|---|---|---|
| tiny | 6.88% | 23.20% | 127.7s | 139.8s |
| base | 5.33% | 19.87% | 303.0s | 285.5s |
| small | 3.00% | 14.13% | 986.2s | 1038.3s |
Audio condition matters more than model size
The clearest pattern in the results: every model's error rate was 3-4x higher on test-other than on test-clean — a far larger gap than the difference between any two model sizes on the same condition. Even Whisper small's error rate on hard audio (14.13%) was still worse than tiny's error rate on clean audio (6.88%). No amount of model upgrading closed that gap.
Diminishing returns on model size
Moving from tiny to small roughly halved the error rate, but inference time grew by nearly 8x over the same range — a return that shrinks sharply as model size increases. The tiny→base step bought a meaningful accuracy gain for a proportionally smaller time cost; the base→small step cost far more time for a comparatively similar-sized accuracy gain.