namu
namu-ai.orgHear Aïr
Active

Maʼauni

Honest measurement for Hausa and Zarma speech

Read the questionsSee progressCollaborate

When is a Hausa word error rate worth trusting?

Test sets, text normalisation, listening protocols and benchmarks that make a Hausa or Zarma result mean something. A word error rate for Hausa can change without the model changing at all.

Test sets

Frozen, speaker-disjoint, with their cells and limits stated.

Normalisation

Strict and folded forms, published with test cases.

Listening

Blind, multi-rater protocols with agreement published.

A word error rate for Hausa can change without the model changing at all. One scorer treats the apostrophe in saʼa as a space. Another deletes numbers. A transcript writes kasa where the speaker said ƙasa. A test set that is 620 of 621 women cannot tell you how a model hears men, and one recorded in Nigeria cannot tell you how it hears Niger. Most published Hausa numbers come without these conditions, so they cannot be compared, and they can flatter.

Maʼauni is our measurement programme. We will publish a versioned Hausa text normaliser, with test cases, that keeps the hooked letters ɓ ɗ ƙ ƴ as the distinct sounds they are. Every result we report will come twice, strict and folded, so readers can see how much depends on spelling. We are building speaker-disjoint test sets that say which dialects, genders and phone conditions they contain, and which claims they cannot support. We run blind, randomised listening tests with native speakers, and we will report how much the listeners agreed.

A number we publish will always come with its conditions. When we cannot measure something, we will say so.

What we do not claim yet

  • We present standards and tools, not a leaderboard.
  • Our listening studies so far have too few raters to report agreement; we will not describe them as validated.
  • Internal measurements appear only with their conditions, after review.

Strict and folded One transcript scored two ways, computed in your browser with word-level alignment.

Folding hooked letters halves this error rate and hides the mistake a Hausa reader notices first.

QuestionsWhat this programme sets out to answer

  1. 01

    What is a correct, versioned Hausa and Zarma text normaliser for scoring, and how much do published results move between toolchains on the same outputs?

  2. 02

    What test sets would support dialect, gender and channel claims for Niger and Nigeria, and how small can each cell be?

  3. 03

    How should tone and vowel length be evaluated in synthesised speech when the spelling does not mark them?

  4. 04

    How reliable are automatic judges against native listeners for Hausa?

ProgressDates appear only on finished steps

  1. Done · Oct 2026Hausa normaliser in codeStrict and folded forms, apostrophe unification, hooked-letter repair. Internal; v1 to be published.
  2. Done · Oct 2026Blind listening-test toolchainRandomised rating pages with agreement statistics. Internal.
  3. In progressNormaliser v1Published with test cases; this site’s explainer will run on the same code.
  4. NextRe-scoring noteOpen Hausa models on public test sets under several normalisers, strict and folded.
  5. NextListening protocol v1Multi-rater, dialect-recorded, pre-registered; agreement published.
  6. NextTest set v1From Rumbu, with its cells and suppression rules stated.
  7. NextBenchmarksCodec and full-duplex, each with an evaluation card.

What we will release

  • Maʼauni normaliser v1 (Hausa and Zarma), with a web port and test cases
  • A consented test set with a data statement
  • An open listening-test kit
  • Paper: how much is a Hausa WER worth?

Work with us on Maʼauni

We are looking for native listeners across Hausa dialects, and researchers who want to re-score their Hausa results with a shared normaliser.

Get in touch

Related reading

More research

All research

Hira

Speech models that listen while they speak, built for Hausa and judged against how Hausa speakers actually take turns.

Rumbu

Hausa speech collected with consent and fair pay, from the people and places public datasets leave out.

Zarma

The first open, consented speech resources and honest baselines for Zarma, spoken by about six million people along the Niger.

Tankaɗe

The methods that turn radio archives, field recordings and crowd audio into Hausa and Zarma data a model can learn from and a reader can trust.