In ten days, a small Māori radio station in New Zealand got two and a half thousand people to read their language aloud to a computer. Why chatbots sound brilliant in some languages and clumsy in others, what that costs you, and a two-language test to find out for yourself.
Takeaways
- A chatbot reads tokens, pieces from a vocabulary built from its training text: in one 2023 Oxford study, the same sentences took about 3 times as many tokens in Arabic as in English with the ChatGPT/GPT-4 tokenizer, and up to 15 times in Shan.
- More pieces can mean a double tax: in a 2023 study of 22 languages using the model behind ChatGPT through OpenAI's paid developer service, speakers of many languages paid more for the same work and often got poorer results.
- This week: run one task in two languages, set your own rule for which language to use for which job, and keep a fluent human as the checker.
Chapters
- Ten days in Kaitaia0:17
- The same sentence, fifteen times2:27
- Pay more, get less4:27
- Two speakers, one machine7:12
- Hear it done8:25
- Build yours9:44
- The answer11:22
References
- Hao, K. — A new vision of artificial intelligence for the people (MIT Technology Review)(2022)
- Te Hiku Media — Hopu Kōrero Māori Group Competition(2018)
- NVIDIA — Māori speech AI model helps preserve and promote New Zealand Indigenous language(2024)
- Olin College of Engineering — Olin alum using AI to preserve te reo Māori(2023)
- Petrov, A., La Malfa, E., Torr, P. H. S., & Bibi, A. — Language model tokenizers introduce unfairness between languages (NeurIPS 2023)(2023)
- Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D. R., Smith, N. A., & Tsvetkov, Y. — Do all languages cost the same? Tokenization in the era of commercial language models (EMNLP 2023)(2023)
- Common Crawl — Statistics of Common Crawl monthly archives: languages (CC-MAIN-2026-39)(2026)