How we use cookies
Telonic

ResearchAnthony Ombrore

Testing agents on conversations that switch language mid-sentence

Gulf customers put English words inside Arabic sentences as a matter of course, and the published evidence says speech systems fail on those conversations for reasons the standard accuracy score cannot see. This article sets out what is known about the switching, where systems break, and how a test has to be built before it tells you anything.

On this page
  1. The situation
  2. What the evidence shows
  3. What that means
  4. How we design for it
  5. What we don't know yet
  6. Sources

The situation

A buyer messages a developer's customer care line on a Thursday evening:

هلا، بغيت أعرف الـ next instalment حق الـ unit، متى الـ due date؟

Hello, I want to know the next instalment for the unit, when is the due date? The sentence is Gulf Arabic. The three words that carry the request are English. On a construction-linked payment plan the answer depends on which milestone the project has reached, not on a calendar date, so the agent has to identify the unit, look up the milestone and reply in a way the buyer can act on.

Nothing about the message is unusual. Testing for it is. Most published measures of how well a system handles Arabic are taken on Arabic alone, on English alone, or on formal Arabic that nobody uses on a service line. So the questions for anyone buying or building an agent for the UAE or Saudi Arabia are how common this mixing is, why it breaks speech and language systems, and how to test so that it is covered rather than assumed.

What the evidence shows

How common it is, and what shape it takes

Linguists separate two kinds of switching. Inter-sentential switching changes language between sentences. Intra-sentential switching changes language inside a sentence, as in the message above. The distinction dates from foundational work in 1980 [24], and it matters here because the two put a system under different stress.

The Gulf evidence points firmly to the second kind. Mixat, the largest public corpus of Emirati speech with English mixed in, transcribed 14.9 hours of two podcasts. 36% of sentences (1,947 of 5,316) contained a switch, 61% were Arabic only and 2% were English only. Its code-mixing index, a score from 0 to 1 that rises as a sentence becomes more evenly split between languages, averaged 0.11 across the switched sentences: Arabic sentences with a few English words in them [1]. A second Emirati corpus, 12 hours of Zoom role-plays by 16 university students, found 19% of utterances switched, and English nouns 56.9% more frequent in switched utterances than in monolingual ones [2]. The comparable Egyptian corpus, 12 hours of interviews with 38 Cairo students, found 63.2% of sentences mixed, 88.9% of them Arabic with English embedded, and 66.3% of the English stretches a single word long. The word most often immediately before a switch was the Arabic definite article [3], which is exactly the pattern in the message above. The one Saudi corpus, 4.45 hours from a single podcast with four male speakers, shows the same shape, with English at roughly one token in five [4].

These corpora are podcasts, interviews and role-plays by young, educated speakers, so they show how switching looks, not how often the whole population does it. A 2019 survey of 100 Emirati university students found 78% said they mixed the two languages within sentences [6]. A 2024 survey of 566 cash machine users in Dubai gives the other side: 99.7% of non-Arab users chose English, Emiratis chose English only 19.7% of the time, and Arab expatriates in higher-paid jobs chose English 69% of the time while those in lower-paid jobs chose Arabic 84.3% of the time [5]. In Saudi Arabia the 2022 census counted 58.4% of 32.2 million residents as Saudi nationals [7]. An agent serving the region will meet English-only, Arabic-only and mixed conversations, and which one arrives depends on who the customer is.

Two layers sit under the switching. The first is dialect. Arabic is diglossic: the formal variety, Modern Standard Arabic, is written and used for formal speech, while everyday speech is in a regional dialect [23]. A Gulf caller speaks Gulf Arabic, much of the region's workforce speaks Egyptian or Levantine, and none of them speaks Modern Standard Arabic to a service line. The second is Arabizi, Arabic typed in Latin letters with numerals standing in for sounds Latin has no letter for, such as 3 for ع and 7 for ح. It was documented among Zayed University students in 2003 [25]. In the 2019 survey, 45% of students said they never used it and 16% said they always did [6], so a WhatsApp thread can contain Arabic script, Latin script and both.

Where the systems break

The working assumption in this research was that failures cluster at the switch point. The evidence says the switch is one of three failures, and not the largest.

The largest is dialect. Speech recognition is measured by word error rate: the share of words in a transcript that are wrong, missing or inserted compared with what was said. Lower is better, and a figure above 100% means the system produced more wrong words than there were words. On a Saudi broadcast test set, one widely used open speech model scored 27.95% on Modern Standard Arabic and 59.92% on Gulf dialect, with Egyptian at 59.28% [8]. That gap has nothing to do with English. It is the price of a model trained mostly on formal Arabic meeting a customer who does not speak it.

The second failure is doing the wrong task. When the Mixat authors ran a general-purpose speech model over their Emirati recordings, it scored 12.06% on the English-only sentences and 195.98% on the Arabic-only ones. The model "translates the sentences into MSA, and the translations are often correct. This shows that Whisper in fact recognizes the dialect and the code-switching, but performs the wrong task" [1]. An eight-dialect study saw the same on its mixed segments: "Whisper failed to produce any code-switched words in all scenarios. Notably, even when the decoding language was set to English, Whisper performed a translation task even when specifying the task as 'transcription'" [9]. The same behaviour is documented on Mandarin-English speech [26]. A customer says "claim" and the transcript holds the Arabic word for claim, or the whole sentence comes back in formal Arabic the customer never spoke.

The third failure is the switch itself. It is real, but mostly invisible in the headline score. A 2025 study introduced a measure that scores only the switched words. Training a model on more monolingual Arabic and English improved its overall word error rate on mixed test sets "while the recognition of actual code-switched instances is deteriorating" [10]. A model can get better on average and worse on the product names, reference numbers and technical terms that carry the request, and the usual test reports an improvement.

The major cloud speech services say as much. One states that its continuous language detection "doesn't support changing languages within the same sentence. For example, if you're primarily speaking Spanish and insert some English words, it doesn't detect the language change per word" [15]. Another allows up to three candidate languages, transcribes "using the best-fit language from the list", and calls the feature "ideal for apps that need to transcribe short statements like voice commands or search" [16]. These are honest descriptions of systems built to pick one language per utterance.

Why the usual score misleads

Word error rate has a specific problem with mixed speech. If a customer says "booking" and the system writes it in Arabic letters, a reader understands it and the metric counts it wrong. A 2022 study had four bilingual annotators post-edit 3,903 transcripts of Egyptian mixed speech, then asked which automatic score best tracked how much editing a human needed. Plain word error rate correlated at 0.55. Character error rate after transliterating English words into Arabic script and normalising spelling correlated at 0.80 [11]. A 2024 method built on Emirati data accepts an exact match, a transliteration or a translation of a switched word as correct [12]. Without an adjustment of this kind, a test set of Gulf conversations penalises correct behaviour and rewards the wrong-task failure whenever it matches the script of the reference.

The deeper point is that transcription is not the outcome. A 2021 study built pairs of transcript sets with identical word error rates, 6.16% in both, and found that the set whose errors preserved meaning gave 96.22% intent accuracy, the share of requests where the system understood what was wanted, against 94.63% for the set whose errors did not [13]. On a 58-hour English benchmark of spoken requests to a home device, the best system understood the intent only about 5 points less often on real audio at 16% word error rate than on perfect transcripts, but its accuracy on the details in the request, names, dates and places, fell from 78.19 to 69.53 on the benchmark's own measure [14]. Intent is robust to transcription errors. Details are not. In a claims call the intent is "where is my claim"; the detail is the plate number. Both need testing, separately. Both studies are in English; no published study measures intent accuracy on Arabic-English mixed speech, which is a gap, not a reassurance.

Speaking the reply

A voice agent has to say a mixed sentence as well as hear one. Text-to-speech, the system that turns the written reply into audio, has its own switch problem. One major provider states that its tag for marking a foreign-language span inside a sentence is "not supported" for "Semitic languages such as Arabic, Hebrew, and Persian" and "will result in silence" [17]. The first published work on Arabic text-to-speech that handles dialect and mixing appeared in 2025, on one hour of Egyptian-English speech from a single speaker, and it handled mixing by splitting each sentence into monolingual phrases and synthesising them separately [18]. Nothing equivalent exists for Gulf Arabic. Numbers add a layer: Arabic numerals inflect for gender, and dialects drop those markers but read numbers in their own ways, so a reference number or an amount in dirhams can be right on screen and wrong out loud [19].

Automatic language detection is also less settled than it sounds. One voice agent platform warns that "background speech or an unusual accent can occasionally trigger a switch the caller did not intend", and offers a setting that confines switching to the first two turns [27]. Detection has to be tested for false switches, not only correct ones.

What is not being tested

A 2025 survey found word error rates of 24.8% to 53.8% across the mixed Arabic speech corpora that exist and states that "CSW Arabic remains underrepresented in current benchmarks" [20]. A separate 2025 survey of more than 40 Arabic language model benchmarks lists code-switching evaluation and multi-turn dialogue as critical gaps [21]. The public Gulf material for testing mixed speech totals a few tens of hours of podcasts, television and student role-plays, mostly under licences that forbid commercial use, and none of it recorded on a phone line to a business. There is no public Gulf test set of mixed-language customer requests labelled with what the customer wanted.

One finding runs the other way. On written text, a 2025 study of large language models found that inserting English words into a non-English sentence "often improves comprehension, especially for models with limited proficiency in the matrix language", while inserting foreign words into English degraded it [22]. The English word in an Arabic WhatsApp message may help the model. The spoken version of the same word is where the difficulty concentrates.

What that means

The hypothesis we started with was that failures cluster at switch points. The evidence corrects it. Failures cluster first on dialect, second on the model doing a different task from the one asked, and only third on the switched words, which the standard score hides. A test built only around switch points would miss the two larger failures. A test built on single-language accuracy would miss all three.

The trade-offs are worth knowing. Constraining a speech service to one language raises its accuracy on that language and removes its ability to follow a switch. Forcing a general model into Arabic makes it translate. Accepting transliterated words as correct makes the score honest and makes it incomparable with a vendor figure that did not. And better average accuracy can coexist with worse accuracy on the words that matter.

So a test of an agent for this region has to hold four things separately: whether the Arabic frame was understood in the dialect the customer used; whether the English inserts came through; whether the intent and the details were captured, scored as understanding rather than as transcript; and whether the reply was spoken correctly, numbers included. It has to use a metric that accepts a correct word in either script. And it has to be built from the conversations the business actually receives, because no public data set stands in for them.

How we design for it

Our agents follow a speaker who mixes Arabic and English in the same sentence, and they work out which language the customer is using from the first few words rather than asking them to choose. They understand and reply in Gulf, Egyptian and Levantine Arabic, Modern Standard Arabic and English, because a Dubai or Riyadh customer base is all of these, and the reply follows the dialect the customer used.

Where the agent is unsure it asks a clarifying question rather than guessing, and a conversation it cannot resolve is transferred to a person. Because the underlying speech and language models differ so much by dialect, models are selected per deployment and can be changed, so a client can see what a change does before customers do.

The test design above is how our evaluation sets are built: test conversations drawn from real patterns in each industry, scored on whether the agent understood what the customer wanted and captured the details correctly, the unit number, the claim reference, the flight, rather than on the transcript alone, and checked against what a resolved outcome means in a claim or a handover. Language preference memory means a customer who chose English last time is greeted in English this time, and one who wrote in Gulf Arabic is not answered in formal Arabic.

We do not publish accuracy figures for our own agents. When we do, they will come with the test set, the dialect mix and the scoring method, because without those a number for mixed-language speech means nothing.

What we don't know yet

Nobody has measured how often customers switch language on Gulf service lines, or what share of those conversations are Arabic, English or mixed. The corpora above are the best available and none was recorded on a phone line to a business.

No published Arabic study measures error at the switch point itself, or on English product names, reference numbers and spoken digits inside Arabic sentences. The mechanism is documented; its size in a customer conversation is not.

No independent evaluation exists of how commercial text-to-speech voices handle a Gulf Arabic sentence with English inside it. Vendor documentation describes limits; nobody outside the vendors has measured them.

There is no evidence either way on whether Arabizi use is falling as Arabic keyboards become universal, and no study of whether Gulf listeners find Egyptian or Levantine replies acceptable in a service call.

Each of these is a question a test set built from a business's own conversations can begin to answer. That is the reason to build one before deployment rather than after.

Sources

  1. 1.Al Ali, M. K. and Aldarmaki, H. Mixat: A Data Set of Bilingual Emirati-English Speech. SIGUL workshop at LREC-COLING, 2024.⁠https://aclanthology.org/2024.sigul-1.26/
  2. 2.Hamed, I., Eryani, F., Palfreyman, D. and Habash, N. ZAEBUC-Spoken: A Multilingual Multidialectal Arabic-English Speech Corpus. LREC-COLING, 2024.⁠https://aclanthology.org/2024.lrec-main.1546/
  3. 3.Hamed, I., Vu, N. T. and Abdennadher, S. ArzEn: A Speech Corpus for Code-switched Egyptian Arabic-English. LREC, 2020.⁠https://aclanthology.org/2020.lrec-1.523/
  4. 4.Alharbi, S. et al. (SDAIA National Center for AI). Saudilang Code-Switch Corpus, 2024.⁠https://huggingface.co/datasets/SDAIANCAI/Saudilang-Code-Switch-Corpus
  5. 5.Al-Issa and Sulieman. Language choice at ATMs in Dubai. Frontiers in Communication 9:1355632, 2024.⁠https://www.frontiersin.org/journals/communication/articles/10.3389/fcomm.2024.1355632/full
  6. 6.Hopkyns, Zoghbor and Hassall. The use of English and linguistic hybridity among Emirati millennials. World Englishes, 2020.⁠https://doi.org/10.1111/weng.12506
  7. 7.General Authority for Statistics, Saudi Arabia. Saudi Census 2022 results, via Saudi Press Agency, 31 May 2023.⁠https://www.spa.gov.sa/w1911463
  8. 8.Wang, Alhmoud and Alqurishi (Elm). Open Universal Arabic ASR Leaderboard. Interspeech, 2025.⁠https://www.isca-archive.org/interspeech_2025/wang25_interspeech.pdf
  9. 9.Talafha et al. Casablanca: Data and Models for Multidialectal Arabic Speech Recognition. EMNLP, 2024.⁠https://aclanthology.org/2024.emnlp-main.1211/
  10. 10.Ugan, E. Y., Pham, N. Q., Bärmann, L. and Waibel, A. PIER: A Novel Metric for Evaluating What Matters in Code-Switching. ICASSP, 2025.⁠https://arxiv.org/abs/2501.09512
  11. 11.Hamed, I., Hussein, A., Chellah, O., Chowdhury, S., Mubarak, H., Sitaram, S., Habash, N. and Ali, A. Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition. IEEE SLT, 2022.⁠https://arxiv.org/abs/2211.16319
  12. 12.Kadaoui, K., Al Ali, M., Toyin, H., Mohammed, I. and Aldarmaki, H. PolyWER: A Holistic Evaluation Framework for Code-Switched Speech Recognition. Findings of EMNLP, 2024.⁠https://aclanthology.org/2024.findings-emnlp.356/
  13. 13.Kim, S., Arora, A., Le, D., Yeh, C.-F., Fuegen, C., Kalinli, O. and Seltzer, M. L. Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding. Interspeech, 2021.⁠https://www.isca-archive.org/interspeech_2021/kim21e_interspeech.pdf
  14. 14.Bastianelli, E., Vanzo, A., Swietojanski, P. and Rieser, V. SLURP: A Spoken Language Understanding Resource Package. EMNLP, 2020.⁠https://aclanthology.org/2020.emnlp-main.588/
  15. 15.Microsoft. Language identification with the Speech service. Azure AI documentation, updated 4 March 2026.⁠https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-identification
  16. 16.Google Cloud. Detect language in audio (multiple languages). Speech-to-Text documentation, updated 24 September 2026.⁠https://docs.cloud.google.com/speech-to-text/docs/multiple-languages
  17. 17.Google Cloud. Speech Synthesis Markup Language (SSML). Text-to-Speech documentation, updated 24 September 2026.⁠https://docs.cloud.google.com/text-to-speech/docs/ssml
  18. 18.Lodagala, V. S., Alkanhal, M., Izham, M., Mehta, S., Chowdhury, S. A., Makki, M., Hussein, A., Henter, G. E. and Ali, A. SawtArabi: A Benchmark Corpus for Arabic TTS. Standard, Dialectal and Code-Switching. Interspeech, 2025.⁠https://www.isca-archive.org/interspeech_2025/lodagala25_interspeech.pdf
  19. 19.Albasiri, E., Kim, M., Ferchichi, N. and Olabiyi, O. Inverse Text Normalization for Arabic Numbers in Streaming ASR. CAWL workshop at LREC, 2026.⁠https://aclanthology.org/2026.cawl-1.11/
  20. 20.Hamed, I., Sabty, C., Abdennadher, S., Vu, N. T., Solorio, T. and Habash, N. A Survey of Code-switched Arabic NLP: Progress, Challenges, and Future Directions. COLING, 2025.⁠https://aclanthology.org/2025.coling-main.307/
  21. 21.Alzubaidi et al. A survey of Arabic large language model benchmarks, 2025 (preprint).⁠https://arxiv.org/abs/2510.13430
  22. 22.Mohamed, A., Zhang, Y., Vazirgiannis, M. and Shang, G. Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text, 2025 (preprint).⁠https://arxiv.org/abs/2506.14012
  23. 23.Ferguson, C. A. Diglossia. Word 15(2), 1959.⁠https://doi.org/10.1080/00437956.1959.11659702
  24. 24.Poplack, S. Sometimes I'll start a sentence in Spanish y termino en español: toward a typology of code-switching. Linguistics 18(7-8), 1980.⁠https://doi.org/10.1515/ling.1980.18.7-8.581
  25. 25.Palfreyman, D. and Al Khalil, M. "A Funky Language for Teenzz to Use": Representing Gulf Arabic in Instant Messaging. Journal of Computer-Mediated Communication 9(1), 2003.⁠https://doi.org/10.1111/j.1083-6101.2003.tb00355.x
  26. 26.Peng, P., Yan, B., Watanabe, S. and Harwath, D. Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization. Interspeech, 2023.⁠https://par.nsf.gov/servlets/purl/10478271
  27. 27.ElevenLabs. Language detection (Agents platform documentation).⁠https://elevenlabs.io/docs/agents-platform/customization/tools/system-tools/language-detection

Get new research as it's published

Occasional emails when we publish.

Subscribe

Read next