- User
- Model reply
- Short reaction
- Starts over the user
Toward spoken conversation in Hausa
A full-duplex model learned timing but not answers. A frozen language model with a trained ear and mouth answers 12 probe questions in about one second, but the live system hears through a transcript and is not end to end.
What we found
Duplex
One duplex model learned timing first
On about 50 hours of synthetic dialogue, the full-duplex model learned to take its turn within 500 steps; on 40 other held-out dialogues it answered 95.6% at the last step.
Takes its turn
% of the user’s turn ends answered
Answers what was asked
relevance, 1 to 5, from an automatic judge that failed calibration: read it as a trend
training steps on about 50 hours of synthetic dialogue
+0.22 nats per word
Muting the user’s channel raised its loss on held-out replies by only this much. It barely used what it heard.
Spoken probe questions answered on topic, of 12
a multilingual language model inside the duplex model, read by hand: about 5
that language model alone, reading them as text: 12
Its replies stayed fragments: an uncalibrated automatic judge rated them 2.7 out of 5, against 4.4 for scripted replies, and our one native listener also rated such replies lower. Muting the user's channel raised its loss on held-out replies by only 0.22 nats per word: the model barely used what it heard. With an 8B multilingual language model inside, it answered about 5 of 12 spoken questions on topic, as read by hand; the language model alone, as text, answered all 12.
Ear
An ear lets a frozen model hear
We froze a 9B multilingual language model and built an ear that turns speech into soft tokens in its input space, and a mouth that speaks its text.
The live system hears this way: a public recogniser writes the user’s words and the language model reads the text. It is not end to end.
a character hearda character misheardreaction gap: how differently the language model reacts than to the true words (lower is closer). Each travelling mark stands for a character; the share drawn misheard follows the route’s character error rate, not which characters.
On one speaker's phone speech, the language model reacted to the ear's vectors about as it did to a public recogniser's transcript: a reaction gap of 0.296 against 0.295. But the ear still misheard more, 14.7 against 9.5% character error rate, so the live system hears through a transcript and is not end to end.
Timing
About one second, by rules
A noise floor estimated over a recent window let the live recogniser find the end of the user's speech after 0.50 seconds instead of 0.96.
*6 of 16 two-sentence turns were still answered early, in simulation
End of speech: medians on 60 clips through the recogniser alone, whisker to the 90th percentile. Reply delay: medians on the 12 probe questions, from the user’s last loud frame to the first sound. Hardware and other settings also changed between the two.
With a faster engine as well, the median reply delay on 12 probe questions fell from about two seconds to 0.88 to 0.96, with no talk-over. Turn-taking here is a hand-set policy. Text alone cannot tell that a second sentence follows, and 6 of 16 two-sentence turns were still answered early.
Care
Style training removed a pointer to help
A small style adapter cut replies over 30 words from 37% to 15% and written-style markers from 31.5% to 1.9%, keeping 19 of 21 keyword hits.
The care route asks the unadapted model one yes-or-no question about the last messages and sends distress turns back to it. It adds about 24 ms.
distress turn, pointed to helpdistress turn, no pointerordinary turnOne mark per turn: the 8 distress turns of the 24-item safety probe and 339 ordinary turns. Their order, and which distress turns got a pointer, are not in the data; the counts are. Shares are of replies to the 54-item question set; a dot in the sheet means not measured with the route.
Unadapted Style adapter Care route Replies over 30 words 37% 15% not measured Written-style marker 31.5% 1.9% not measured Keyword hits, of 21 18 19 19 Distress turns pointed to help, of 8 5 4 5 First word, median 115 ms 170 ms 194 ms On our 24-item safety probe it pointed people in distress to outside help in 4 of 8 cases, against 5 of 8; its training data held no distress conversation. A care route sends such turns to the unadapted model: it routed 7 of 8 distress turns and none of 339 ordinary ones, adding about 24 ms.
What comes next
- Train the ear on real phone speech, so that its hearing beats the transcript and the live system can hear without text again.
- Use pitch at the pause, or the ear's vectors, for turns of two sentences.
- Record more natural speech of the consenting speaker for the mouth.
- Run a native listening panel with decision rules fixed in advance.
- Have native speakers, with clinical guidance, write care lines and distress probes before any public demonstration.
- Apply the same structure to Zarma and other languages of the Sahel.
The paper
Toward Spoken Conversation in Hausa: An Exploration
Namu AI LabNovember 202615 pagesPDF
Abstract
We report an exploration of spoken conversation in Hausa: a system that hears a person speak Hausa and answers aloud in Hausa, with the timing of a conversation. We tried three designs in turn and measured each one. A full-duplex speech-to-speech model adapted to Hausa learned the form of conversation from about 50 hours of synthetic two-channel dialogue: it answered 95.6% of the turn ends in 40 held-out synthetic dialogues with a median delay of 0.78 seconds, in the voice of a consenting team member. It did not learn the content. Its replies barely depended on what the user said; an uncalibrated automatic judge rated them 2.7 out of 5 against 4.4 for scripted replies, and our one native listener also rated such replies below scripted ones. Placing a multilingual language model inside the same duplex model raised on-topic answers to our 12 spoken probe questions only to about 5 (read by hand), although that language model alone answered all 12 sensibly as text. We therefore kept a multilingual language model frozen and built around it an ear, which turns speech into soft tokens in the language model's own input space, and a mouth, which speaks the model's text in the team member's voice. Through the ear, the language model reacted to one speaker's phone speech about as it did to a public recogniser's transcript (a reaction gap of 0.296 against 0.295), but the ear misheard more (14.7 against 9.5% character error rate), so the live system still hears through the recogniser's transcript and is not end to end. Its turn-taking is a hand-set policy around a learned turn-end reader; on 12 probe questions it answered after about one second and did not talk over the user. A spoken-style adapter shortened the language model's replies but weakened its answers to people in distress, until a care route sent such turns back to the unadapted model. All results are automatic measurements or the judgements of one native listener, the team member whose voice the system uses.
Give this research more voices
What comes next needs consented speech from many more speakers. If you speak Hausa, you can record short missions from your phone.
