How we measure transcription accuracy

How Autrans measures word error rate language by language: the public test recordings, how the interval is built, and what the figures cannot tell you.

What a word error rate is, which recordings we test on, how sure we are, and what the numbers cannot tell you about your own audio.

Test Autrans on your own recording

The short version:

  1. We measure word error rate, language by language, on public test recordings. Every figure names the kind of speech and the test set.
  2. Every figure today is read speech. English and Nigerian Pidgin have none.
  3. The honest test is your own audio, and the free minutes are there for it.

What word error rate means

Take a reference written by a person and a transcript written by Autrans. Count the words that were swapped, missed or added, and divide by the number of words in the reference. Lower is better.

The rate can pass 100 when a transcript adds many words, which is why we do not turn it into an "accuracy" percentage. A sentence worked through step by step is on what word error rate means.

Before scoring, both sides are treated the same way: one composed form of each letter, lowercase, no punctuation, tidy spacing. A short list of Nigerian spelling variants, such as wetin and wettin, counts as one word, so a correct word is not penalised for a spelling choice.

Which recordings we use

We score against public recordings that come with a transcript written by people, so nobody has to take our word for the reference. FLEURS, from Google, covers 102 languages with the same sentences in each. NCHLT, from South Africa's SADiLaR, covers four of ours. BibleTTS Asante Twi covers Twi. For FLEURS we score 250 recordings from the start of the test file, or all of them when there are fewer.

LanguageTest setRecordingsStatement
AfrikaansFLEURS250Word error rate 21.1% on read speech, FLEURS test set, measured Sep 2026.
AmharicFLEURS250Word error rate 24.3% on read speech, FLEURS test set, measured Sep 2026.
ArabicFLEURS (Egyptian Arabic)221Word error rate 13.7% on read speech, FLEURS test set, measured Sep 2026.
FrenchFLEURS (Parisian French)222Word error rate 5.3% on read speech, FLEURS test set, measured Sep 2026.
HausaFLEURS250Word error rate 31.8% on read speech, FLEURS test set, measured Sep 2026.
IgboFLEURS250Word error rate 77.4% on read speech, FLEURS test set, measured Sep 2026.
isiXhosaFLEURS250Word error rate 52.2% on read speech, FLEURS test set, measured Sep 2026.
isiZuluFLEURS250Word error rate 39.6% on read speech, FLEURS test set, measured Sep 2026.
LingalaFLEURS250Word error rate 37.5% on read speech, FLEURS test set, measured Sep 2026.
OromoFLEURS41Word error rate 79.7% on read speech, FLEURS test set, measured Sep 2026.
PortugueseFLEURS (Brazilian Portuguese)215Word error rate 3.5% on read speech, FLEURS test set, measured Sep 2026.
SesothoNCHLT250Word error rate 69.4% on read speech, NCHLT test set, measured Sep 2026.
SetswanaNCHLT250Word error rate 62.2% on read speech, NCHLT test set, measured Sep 2026.
ShonaFLEURS250Word error rate 47.3% on read speech, FLEURS test set, measured Sep 2026.
SomaliFLEURS250Word error rate 49.5% on read speech, FLEURS test set, measured Sep 2026.
SpanishFLEURS (Latin-American Spanish)199Word error rate 2.2% on read speech, FLEURS test set, measured Sep 2026.
SwahiliFLEURS250Word error rate 14.0% on read speech, FLEURS test set, measured Sep 2026.
TshivenḓaNCHLT252Word error rate 74.0% on read speech, NCHLT test set, measured Sep 2026.
TwiBibleTTS (one narrator)250Not yet measured on a public test set.
WolofFLEURS250Word error rate 58.0% on read speech, FLEURS test set, measured Sep 2026.
XitsongaNCHLT250Word error rate 71.1% on read speech, NCHLT test set, measured Sep 2026.
YorubaFLEURS250Word error rate 68.3% on read speech, FLEURS test set, measured Sep 2026.
EnglishNoneNoneNot yet measured on a public test set.
Nigerian PidginNoneNoneNot yet measured on a public test set.

The largest open collection of Yoruba, Hausa and Igbo speech we know of is licensed for non-commercial use only, so we do not score against it.

How sure we are

Each figure comes with a 95 per cent interval, built by resampling the test recordings many times. We resample whole sentences or whole speakers, because recordings from the same sentence or speaker are not independent. In FLEURS several people read the same sentence, so Igbo's 250 recordings are 88 different sentences. Where no grouping works, the interval is too narrow to trust. Oromo has 41 recordings and Twi is one narrator, so both carry a warning beside the figure.

Marks and spelling

Some languages lose a lot of the score to marks on letters, not to misheard words. For those we also show the same recordings with accents, tone and vowel marks removed from both sides, but only when that moves the result by three points or more. Yoruba and Igbo are scored after the engine puts tone marks back. Hausa is not restored, because in testing it made transcripts worse.

What the numbers do not tell you

  • Conversation. Read speech is the easy case, and the same system can score very differently on conversation. We do not publish a conversational figure yet.
  • Your recording. Noise, phone lines, accents and several people talking at once are not in these figures.
  • Other companies' figures. A number from another company is comparable with ours only if both used the same recordings and the same scoring rules. That is rarely true, so we do not compare.
  • African-accented French, Spanish, Portuguese and Arabic. Those four were scored on Parisian French, Latin-American Spanish, Brazilian Portuguese and Egyptian Arabic.
  • Meetings. We have not benchmarked the meeting bot or live meeting notes.

Apart from English and Nigerian Pidgin, which have no figure yet, a language opens only once it has a measured figure, and one we tested and found not good enough stays closed. Every figure on this site is read from one table, copied from the written record of the test run. An automated check fails our build if the two disagree. Figures were measured in September 2026.

Test it on your own recording

  1. Pick a minute of speech you know well and type exactly what is said.
  2. Upload the recording to Autrans, choose its language, and read the transcript.
  3. Count the words that differ: wrong, missing or extra.
  4. Divide by the number of words you typed. Compare it with the statement for your language, and expect a worse result on noisy or crowded audio.

Questions people ask

What is word error rate?

The share of words that differ from a human-written transcript, so lower is better. See what word error rate means.

Is word error rate the same as accuracy?

No. Accuracy is often quoted as 100 minus the error rate, but that breaks when a transcript adds many words, because the rate can pass 100. We report the error rate and do not convert it.

How accurate is Autrans?

It depends on the language and the recording. Find your language in the table, read the statement, then test your own audio.

Related

Start transcribing free

Send Autrans into your next meeting, use it in the room, or upload a recording. Your first 30 minutes are free, given once, and no credit card is needed to start.

Last checked 21 October 2026. Spotted a mistake? Write to hello@autrans.online.