What open language models know of Hausa, Zarma and Fulfulde

What we found

1. Reading

   Zarma costs the most to read

   On the same 150 sentences, a general 9B base model needs 1.14 bits per character for English, 3.17 for Hausa and 4.27 for Zarma.

   General 9B+ 50 African languages

   The chat version of the same model, further trained on 50 African languages. Zarma was not one of them.

   One tick per bit, 100 to a line.

   • English

     96 bits−16%

     English: 114 bits to read 100 characters with the general model, 96 after further training on African languages.

   • Hausa

     148 bits−53%

     Hausa: 317 bits to read 100 characters with the general model, 148 after further training on African languages.

   • Zarmanot among the 50

     334 bits−22%

     Zarma: 427 bits to read 100 characters with the general model, 334 after further training on African languages.

   0100200300400bits to read 100 characters, one tick per bit

   Bits per character × 100, on the same 150 test sentences in each language, spaces included. Thin lines under the ribbons: 95% intervals. Reading cost was not measured for Fulfulde.

   • English1.00×
   • Hausa2.22×
   • Fulfulde2.18×
   • Zarma1.97×

   The chat version of the same model, further trained on 50 African languages, needs 1.48 for Hausa; Zarma, not among them, falls only to 3.34, a 22% cut. The tokeniser is not the cause: it spends 1.97 times the English tokens on Zarma, fewer than the 2.22 it spends on Hausa.

2. Writing

   Understanding Zarma without writing it

   Shown a Zarma sentence, the 9B model further trained on African languages picks its English meaning 77% of the time; matching characters alone gets 58%, and Hausa sentences 100%.

   9B chat9B, 5 examplesTranslation

   0 of 150 written in Zarma

   The 9B chat model further trained on African languages, asked to translate 150 English sentences into Zarma.

   Read the other way: shown a Zarma sentence, the 9B chat model picks its English meaning among four

   character overlap 58%Zarma 77%Hausa 100%

   Measured for the chat model only.

   Other languages: Fulfulde 4, English 3, French 2, others 26.

   Output language by a public text identifier, which recognises 82 of 150 real Zarma sentences as Zarma. Each dot is one sentence; the paper gives only how many went to each language.

   Asked to translate 150 English sentences into Zarma, the same model wrote Hausa 115 times, other languages 35 times and Zarma never. A translation model with a Zarma setting wrote Fulfulde 88 times and Zarma 4 times.

3. Choice

   Choosing a French brain by a fixed rule

   Six open models answered 234 French test turns each inside the same care guard; the mixture-of-experts 26B gave the most good answers, 92%.

   Good answersThe ruleRefusals kept

   Sorted by the rule fixed in advance: fewest failed safety checks, then fewest faults of the brain’s own.

   Good answersSafety checks failed, of 30Own faults

   1. 1
   2. 2
   3. 3
   4. 4
   5. 5
   6. 6

   1. General 9Bchosen

      85%good

      3

      0own faults

   2. Third family 8B

      53%good

      3

      1own fault

   3. Mixture of experts 26B

      92%good

      4

      0own faults

   4. Third family 14B

      58%good

      4

      4own faults

   5. 12B

      90%good

      6

      1own fault

   6. African-continued 9B

      80%good

      13

      10own faults

   failedfailed where the guard overwrote a correct refusalrefusal keptpassed

   Each cell is one safety check; the paper gives how many failed, so their order in a row is arbitrary. Own faults: the brain’s faults before the guard acts, which break ties. The third order is a check made after the choice.

   The rule, fixed in advance, ranks by failed safety checks: the general 9B failed 3 of 30 and was chosen; the African-continued 9B failed 13. Some failures were the guard's: it overwrote correct refusals. Had it kept them, the 26B would fail 2 checks against the 9B's 3 and rank first.

4. Agent

   Translation breaks care before the brain

   In Zarma no open language model we tested writes the language, so the brain reasons in French, with translation in and out, inside a fixed care guard.

   French pivotEnglish pivot

   63 of 111 turns pass every check

   63 of 111 turns pass every check

   The general 9B reasons in French; one translation model carries each turn in and out.

   Recognition

   Into French

   40 first failed here

   Brain in French

   care guard

   Out of French

   Synthesis

   Every check passed

   63

   First failed elsewhere

   8

   The 3 care failures, all in translation, before the brain:

   • a pregnancy → I ate
   • rat poison → spoiled chicken
   • diarrhoea, refusing to drink → an exam

   111 spoken test turns: 86 synthesised, 25 real recordings of one speaker. Each counter is one turn; where it sits in its column is arbitrary. First failed elsewhere: turns whose first failure came at another stage, not broken down in the paper. Care failures by the automatic check; on our reading one more reply opens by minimising.

   Of 111 spoken test turns, mostly synthesised, 63 pass every check with the general 9B in French, against 41 for the English pivot with the African-continued brain. Every remaining care failure happened in translation, before the brain saw the turn.