How to read TTS numbers

The finding that breaks the usual ranking

The standard way to score a speech model is word error rate: synthesise a script, feed the audio back through a speech recogniser, and count the mistakes. Lower is better — or so every leaderboard assumes.

In 2026 a validation study ran sixteen objective metrics against roughly 11,800 human ratings across four speech domains. Word error rate came out negatively correlated with what listeners actually preferred (Spearman ρ ≈ −0.11 to −0.45). Lower error tracked worse perceived quality.

The mechanism is obvious in hindsight. Over-articulated, prosodically flat, recogniser-friendly speech scores beautifully on WER and sounds like a train announcement. The qualities that make a voice sound human — elision, uneven stress, breath, the slight slurring of unstressed syllables — are exactly the qualities that make a recogniser work harder.

A second result from the same study is worth stating plainly: generic speaker-verification embeddings outperformed every purpose-built quality predictor. The models built to predict human opinion did worse than models built to answer a different question entirely.

The six metrics you will meet

RTF — real-time factor. Generation time divided by the length of audio produced. RTF 0.5 means ten seconds of speech took five seconds to make. Below 1.0 is faster than real time; for live conversation you want under about 0.3 so there is headroom for everything else. Always ask which hardware — an RTF without a named GPU is not a number.
Time to first audio. How long between sending text and having something playable. For conversation this matters more than RTF: a system at RTF 0.8 with 90 ms to first audio feels instant, while RTF 0.2 with a two-second wait feels broken. Vendor figures almost always exclude network, queueing and TLS — expect 40–150 ms on top for a real client.
Word error rate. Intelligibility and faithfulness to the script, not beauty — and see above. It also depends entirely on which recogniser was used; two common choices disagree by more than the gap between the systems being compared.
Speaker similarity. How close a cloned voice is to the original. Above 0.90 is excellent, below 0.70 is a different person. Two traps: similarity measured against a codec-reconstructed reference flatters the model, and the embedding model used changes both the scale and the rank order.
Mean opinion score. Human raters scoring 1–5. The gold standard and the most expensive — and not an absolute scale. It depends on the rater pool, the listening conditions, the sentences chosen and which competitors were in the same batch. A 4.2 in one paper and a 4.1 in another tell you nothing.
Predicted MOS. Neural networks trained to guess human ratings. Cheap, reproducible, and easily fooled — they reward clean, smooth, slightly over-processed audio and penalise the rasp and breathiness that make a voice sound alive. Treat a high score as evidence that artefacts are absent, not that quality is present.

Seven ways these numbers mislead

Cross-paper comparison is invalid by default. Different test sets, reference lengths, recognisers and embedding extractors. Only trust numbers produced by one evaluator running every system through the same harness.
Home-field test sets. A model evaluated on the test split of its own training corpus looks better than it is. Shared benchmarks are shared for a reason; a benchmark you have never heard of is usually the authors' own — and the useful question is whether competitors adopted it.
Zero-shot scores understate fine-tuned reality. Published cloning figures use deliberately hostile conditions — short, noisy clips from unseen speakers. With clean studio audio and a fine-tune, expect roughly 0.10–0.15 higher.
Datacentre latency is not your latency. Autoregressive decode is bound by memory bandwidth, not compute, so halve or third a datacentre figure for consumer hardware.
Partial-pipeline throughput. Some benchmarks time the language model and quietly omit the audio decoder. End-to-end is always worse.
Cherry-picked demos. Demo audio is chosen. The honest test is your own text — especially numbers, acronyms, proper nouns, addresses, and anything longer than thirty seconds.
Arena ratings conflate preference with fitness. Blind A/B arenas reward whatever sounds impressive in an eight-second clip. They do not measure consistency across a two-hour audiobook, pronunciation control, or whether the model will read a phone number correctly.

The measurement that matters most, and that nobody publishes

Benchmarks in this field use short utterances. Real work is long. One vendor's own technical report shows speaker similarity collapsing from roughly 0.80 to 0.52 over minutes of continuous generation, with error rates rising fourfold.

Identity drift over long output is a real, measured failure mode that short-utterance benchmarks hide completely — and it is far more decision-relevant than any half-point difference on a clean test set. If you are producing anything longer than a sentence, this is the thing to test yourself.

Three results that say there is no single score

Independently, in the same year, three groups reached the same conclusion from different directions. Objective intelligibility anti-correlates with perceived quality. Control over non-verbal sounds "often decouples from overall speech quality" — a model can top a naturalness arena and be bad at the laughter tag its own demo showcases. And naturalness, expressiveness, identity stability and reliability behave as largely independent axes, concluding that voice systems "should be evaluated as a profile rather than by a single aggregate score."

All three contradict the format of a single ranked list of best models — including, to be explicit, the matrix on this site. That is why it sorts on many columns and ranks on none of them by default.

The test that actually predicts production quality

Synthesise the same 300-word script through every candidate, using your real content including the awkward parts. Listen once at normal speed for naturalness, once at 1.5× for artefacts, and run the output through a recogniser to catch dropped words you will not consciously hear.