Code-mixing: Hinglish and friends
People often mix languages in one sentence and type them in Roman letters. Learn how to handle mixed input, normalise it carefully, and test with real phrases.
In 60 seconds
Code-mixing: Hinglish and friends
People often mix languages in one sentence and type them in Roman letters. Learn how to handle mixed input, normalise it carefully, and test with real phrases.
What mixed input looks like
| Pattern | Example | What makes it hard |
|---|---|---|
| Mixed words | "Mera order cancel kar do" | The model must understand words from two languages together |
| Romanised script | "kya aap delivery date bata sakte ho" | An Indian language written in Roman letters, with no fixed spelling |
| Mixed scripts | "मेरा order अभी तक नहीं आया" | Two scripts inside one sentence |
| Spelling variants | "nahi" / "nahin" / "nai" / "nhi" | Many spellings of the same word |
| Chat style | "plz jaldi bhejo!!! 😡" | Short forms, repeated letters, emoji and punctuation |
Normalising carefully
import re
import unicodedata
# A small, hand-checked list. Real systems learn variants from real user data.
VARIANTS = {
"nahin": "nahi", "nai": "nahi", "nhi": "nahi",
"kyaa": "kya",
"h": "hai",
"pls": "please", "plz": "please",
}
def normalise(text: str) -> str:
text = unicodedata.normalize("NFC", text).lower() # same letters, same code points
text = re.sub(r"(.)\1{2,}", r"\1\1", text) # "bohottt" -> "bohott"
# Remove punctuation and emoji by Unicode category, so Indic vowel signs survive.
text = "".join(" " if unicodedata.category(c)[0] in "PS" else c for c in text)
return " ".join(VARIANTS.get(w, w) for w in text.split())
for msg in ["Mera order abhi tak nahin aaya!!!", "order nhi aaya h, plz check",
"Order NAI aayaaa 😡", "मेरा ऑर्डर नहीं आया"]:
print(f"{msg!r:40} -> {normalise(msg)!r}")- Unicode NFC matters for Indic scripts: some letters can be stored in more than one way (for example a letter with a nukta), and NFC makes both forms the same.
- Removing punctuation by Unicode category keeps Indic vowel signs. Simple patterns like
\w+in Python'srecan cut Indic words into pieces. - The variant list is tiny on purpose. Every entry should be checked by a speaker.
Handling mixed input in an app
- 1
Accept it
Do not force users to pick one language or script. Let them write the way they talk. - 2
Keep the original
Store what the user wrote. Use a normalised copy for search and matching. - 3
Answer in their style
Reply in the language and script the user used, or let them choose. Test that the model does this. - 4
Protect names and numbers
Do not transliterate or "correct" names, order IDs, amounts or phone numbers. - 5
Test with real phrases
Build your test set from real, consented, anonymised user messages, not only textbook sentences.
- Only clean sentences in one script
- Written by the developers in a meeting
- One spelling for every word
- Mixed languages, mixed scripts, emoji and short forms
- Taken from real messages, with personal data removed and permission to use them
- Many spellings of common words, from different regions
Watch and read more
Lab
A 20-phrase code-mixed test set with a normaliser you extended and checked.
The problem
Imagine a support bot for a food delivery app in a city you know.
- Write 20 realistic customer messages the way people really type them: mixed languages, romanised and native script, short forms, emoji. Do not copy real people's messages.
- For each, write the intent (for example: late order, refund, wrong item) and any key facts (order ID, amount).
- Run the normaliser from the lesson on all 20. Add at most five new variants to the list, and check each with a speaker.
- Find at least two messages where normalisation changed or lost something important. Explain what went wrong.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your intent classifier works well on Devanagari Hindi but badly on romanised Hindi. Name two different fixes and one risk of each.Reveal
(1) Transliterate romanised input to Devanagari before classifying. Risk: English words, names and variant spellings may be converted wrongly. (2) Add romanised examples to the training or test data, or use a model that handles both. Risk: you need enough real, varied examples, and must collect them with consent. Test both on a real romanised test set.
Q2Why is it risky to measure a code-mixed chatbot only by how often it replies in the "correct" language?Reveal
Language choice is one small part of quality. A reply can be in the right language and still be wrong, rude, or miss the user's problem. Also, the "correct" language may depend on the user's preference. Measure task success and answer correctness too, and ask users what they prefer.
Questions people ask
Is code-mixing a sign of poor language skills?
No. Researchers who study bilingual speakers describe code-mixing as a normal and often skilful way of speaking. Products should support it, not correct it.
Should my bot reply in Hinglish if the user writes in Hinglish?
Often yes, because it feels natural. But some users prefer a reply in their language's own script, or in English. Test with users and consider letting them choose.
Why not just transliterate everything into the Indian script first?
It can help for some tasks, but it can also go wrong: English words, names and brand names may be converted strangely, and spelling variants give different results. Try it on a real test set and keep the original text.
How can I collect real phrases without invading privacy?
Ask users for permission, remove names, phone numbers, addresses and IDs before anyone reads the data, keep only what you need, and store it securely. The next lesson goes deeper on consent.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.