64 readings run past 30 seconds
Measuring Hausa speech recognition
We measured Hausa speech recognition with public recognisers and public test sets, and found that spelling conventions and the choice of subset move a score about as much as the choice between two recognisers of the same size does on read speech. In the FLEURS Hausa test split, about half of the word types written with a hook also appear without it.
What we found
Spelling
The references disagree with their own spelling
Hausa writes four hooked letters that most keyboards lack. Strict scoring keeps them; folded scoring maps them to b, d, k and y.
written with the hookwritten without itEach word appears as many times as it does in the FLEURS Hausa test references.
The references of four test sets hold from 0.91 to 5.14 hooked letters per 100 words; the broadcast set has 8 ƙ in 12,066 words. In FLEURS, 74 of 147 word types written with a hook also appear without it, and the hooked spelling makes up 46% of their occurrences. For a zero-shot 1B recogniser on FLEURS, hooks explain 13.8% of the strict character edits and word boundaries 16.6%.
FLEURS
Who is in the FLEURS Hausa test set
The public FLEURS Hausa test split holds 621 recordings, 3.34 hours of encyclopaedic sentences read aloud, with no speaker identifiers.
620 recordings are labelled female and one male, so a separate score for men cannot be computed. The 621 recordings read only 331 sentences: 116 once, 140 twice and 75 three times. 64 recordings, a fifth of the audio, run past 30 seconds and 18 past 40; a common 3 to 20 second filter keeps 386.
Recognisers
A decoder told the language stays in Hausa
On one consenting speaker's phone recordings, a 1B recogniser with a decoder told the language reaches 9.5% CER; one with a CTC head, 38.4%.
- read crowd speech3.85.8
- FLEURS6.56.7
- community read speech4.36.7
- spontaneous monologues6.16.7
- phone, one speaker9.538.4
% strict character error · lines: 95% intervals, phone set only
The CTC model wrote 80 of 167 phrases partly in non-Latin scripts, Arabic letters in 64 of them. The decoder wrote none. A public recogniser fine-tuned for Hausa had higher median per-utterance error than a zero-shot multilingual one on all three sets, 7.5 against 3.9% on FLEURS; no intervals.
Forgetting
A Zarma fine-tune forgets the Hausa hooks
Zero-shot, the 1B CTC recogniser writes the reference's hooked letter in 131 of 502 cases on FLEURS; most others become plain letters.
ɓ42
ɗ163
ƙ247
ƴ50
as in the referencehook lostanother letterdropped
Hausa folded character error, FLEURS (483 digit-free utterances up to 40 s)
4.6 %051015Each square is one hooked letter; within a letter the order is arbitrary.
After fine-tuning on Zarma alone it writes one of them, and writes the Zarma letter ŋ for Hausa n 310 times. With Hausa mixed back in, it writes 134 of 502 as in the reference, but different ones: 81 of 163 ɗ and 39 of 247 ƙ, against 39 and 81. Its Hausa folded CER rises from 4.6 to 12.4%. Mixing Hausa back in at about a fifth of utterances keeps it at 5.3% at the checkpoint selected on Zarma.
What comes next
- Native review of the normaliser and of the hooked and plain word pairs.
- A speaker-disjoint Hausa test set that states its dialects, genders and recording conditions, with Hausa from Niger and phone speech.
- Listening tests for tone and vowel length.
- Transcription conventions and evaluation for long spontaneous speech.
- Joint training for recognisers that serve Hausa with its neighbours.
The paper
Measuring Hausa Speech Recognition: Spelling, Test Sets and What a Score Means
Namu AI LabNovember 202614 pagesPDF
Abstract
Hausa is written with four hooked letters that most keyboards lack, ɓ ɗ ƙ ƴ, and an apostrophe that is typed in several ways, while tone and vowel length are not written at all. A Hausa error rate therefore depends on how text is cleaned before scoring and on whose spelling the reference follows. We report what we learned from measuring Hausa speech recognition with public recognisers and public test sets. We describe a text normaliser with three scores: strict, which keeps the hooked letters; folded, which removes hooks and apostrophes; and space-free. The references disagree with themselves: in the public FLEURS Hausa test split, about half of the word types written with a hook also occur without it, and the hooked spelling accounts for 46% of those words' occurrences. The split holds 621 recordings of 331 sentences; 620 are labelled female, none carries a speaker identifier, and 64 are longer than 30 seconds. On read speech, zero-shot multilingual recognisers of about 1B and 3B parameters reach 4 to 8% character error rate (CER). Folded word error rate (WER) is 2.7 to 4.9 times folded CER on every set, except where a recogniser drifts into other scripts. On one speaker's phone recordings, a recogniser with a CTC head reaches 38% CER and writes 80 of 167 phrases partly in non-Latin scripts, while one of the same size with a language-model decoder, told the language, reaches 9.5%. A public recogniser fine-tuned for Hausa has a higher median per-utterance CER than a zero-shot multilingual one on the three sets where we compared them; we computed no intervals for that comparison. Of the 502 hooked letters in 483 FLEURS references, a zero-shot recogniser writes 131 as in the reference. After fine-tuning on Zarma alone it writes 1, and its Hausa folded CER rises from 4.6 to 12.4%; with Hausa mixed back into training it writes 134, and its CER is 5.3% at the checkpoint selected on Zarma. All results are automatic measurements; no native speaker has yet reviewed the normaliser or the spelling counts.
Give this research more voices
What comes next needs consented speech from many more speakers. If you speak Hausa or Zarma, you can record short missions from your phone.
