Why we report every Hausa score twice

A Hausa word error rate depends on how the text is cleaned before scoring. We will report each result strict, with the hooked letters kept, and folded, with them removed.

Word error rate counts the words a recogniser got wrong, as substitutions, deletions and insertions, and divides by the number of words in the reference. Before counting, the scorer cleans both texts. For Hausa, that cleaning step can change the score without any change to the model.

Two scores

Both of our scores start the same way. Capital letters and punctuation are conventions of writing, so we remove them from both texts and write every apostrophe-like mark as one character, ʼ.

Strict scoring then keeps the hooked letters ɓ ɗ ƙ ƴ and the apostrophe. A word written with k where the reference has ƙ counts as an error.

Folded scoring maps ɓ ɗ ƙ ƴ to b d k y and drops the apostrophe before counting.

Folding exists for a fair reason. Hausa is often typed on keyboards without these letters, so many transcripts leave the hooks out or put an apostrophe before a plain letter. If a reference has lost a hook that the model wrote, strict scoring counts the model’s correct spelling as a mistake.

A worked example

The pair below is constructed to show the arithmetic. The second line is not output from any model.

Reference: Ana yin allurar rigakafi kowace Talata da Alhamis, daga ƙarfe takwas na safe.

Hypothesis: ana yin allurar rigakafi kowace talata da alhamis daga karfe takwas da safe

Vaccinations are given every Tuesday and Thursday, from eight o’clock in the morning.

After cleaning, each line has 13 words, and two differences remain. The model wrote karfe without the hook on ƙ, and it wrote da where the reference has na.

Strict scoring counts both, 2 errors in 13 words, a word error rate of 15.4%. Folded scoring turns ƙarfe and karfe into the same word, which leaves 1 error in 13, or 7.7%.

Without any cleaning, the same pair scores 6 errors in 13 words, or 46.2%. Four correct words, Ana, Talata, Alhamis and safe, count as errors because of a capital letter or a punctuation mark.

What folding hides

In this example, folding halves the score. The error it removes is the missing hook, the kind of error a Hausa reader notices first.

In Hausa, ɓ ɗ ƙ ƴ are separate letters for separate sounds. A transcript that writes kasa for ƙasa (land, country) is wrong to a Hausa reader, and folded scoring calls it right. A recogniser that never writes a hooked letter can score well folded and still produce text that a Hausa speaker has to correct by hand.

The apostrophe raises the same question. Writers use several different characters for it. A scorer that does not unify them counts sa'a and saʼa as different words, and one that treats the apostrophe as punctuation splits saʼa in two.

Our rule

Every Hausa word error rate the Lab publishes will come twice, strict and folded, side by side. The gap between the two shows how much of a result depends on spelling. We will name the normaliser and its version.

Ours will be published as Maʼauni normaliser v1, with test cases, and the figure on this page will run on the same code. We have not yet published a Hausa score of our own. When we do, it will follow this rule.

If you report Hausa results and want to re-score them with a shared normaliser, write to us at contact@namuai.org.