AI for Indian languagesMiddleLesson 2377 min read

Speech recognition for Indian languages

Turning speech into text is hard with accents, noise and mixed languages. Learn how word error rate works, write it in Python, and test fairly.

✦ Ask the tutor 🧪 Lab · 35 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Speech recognition for Indian languages

Turning speech into text is hard with accents, noise and mixed languages. Learn how word error rate works, write it in Python, and test fairly.

1/4

In simple words

Imagine taking notes while your aunt speaks on a crowded bus, half in Tamil and half in English, with the horn honking. You will miss some words and mishear others. A speech recognition system has the same problem, and we need a fair way to count its mistakes.

Automatic speech recognition (ASR) turns spoken audio into text. It powers voice search, voice notes to text, call centre tools and helplines. For speech-first users, it is often the front door of an app. If it fails, everything after it fails too.

What makes it hard in India

Accents and dialects. The same language sounds different from region to region and between age groups.
Noise. Traffic, fans, markets, crowded homes, and cheap phone microphones.
Code-mixing. English words inside Indian-language sentences, and the other way round.
Names and numbers. Names of people, villages and schemes; amounts like "two ninety nine" or "do sau".
Spelling choices. Should an English word be written in Roman letters or in the Indian script? Both can be correct.

Word error rate (WER)

The most common measure is word error rate. Compare what the system wrote (the hypothesis) with a careful human transcript (the reference). Count the smallest number of word edits needed to turn one into the other:

Substitutions: a wrong word in place of the right one.
Deletions: a word that was said but is missing.
Insertions: an extra word that was not said.

Then WER = (S + D + I) / N, where N is the number of words in the reference. Lower is better. Because insertions count, WER can be more than 1 (more than 100%).

Word error rate with edit distancepython

def word_error_rate(reference: str, hypothesis: str) -> float:
    """WER = (substitutions + deletions + insertions) / words in reference."""
    ref = reference.split()
    hyp = hypothesis.split()
    if not ref:
        raise ValueError("reference must contain at least one word")
    # d[i][j] = edits to turn the first i reference words into the first j hypothesis words
    d = [[0] * (len(hyp) + 1) for _ in range(len(ref) + 1)]
    for i in range(len(ref) + 1):
        d[i][0] = i                       # delete all i words
    for j in range(len(hyp) + 1):
        d[0][j] = j                       # insert all j words
    for i in range(1, len(ref) + 1):
        for j in range(1, len(hyp) + 1):
            cost = 0 if ref[i - 1] == hyp[j - 1] else 1
            d[i][j] = min(
                d[i - 1][j] + 1,          # deletion
                d[i][j - 1] + 1,          # insertion
                d[i - 1][j - 1] + cost,   # substitution (or match)
            )
    return d[len(ref)][len(hyp)] / len(ref)


pairs = [
    ("mera account balance kitna hai", "mera account balance kitna hai"),
    ("mera account balance kitna hai", "mera acount balance kitna"),
    ("kal subah das baje train hai", "kal subha das bje ki train hai"),
]
for ref, hyp in pairs:
    print(f"WER={word_error_rate(ref, hyp):.2f}  ref='{ref}'  hyp='{hyp}'")

Work the last example by hand: "subah" became "subha" (1 substitution), "baje" became "bje" (1 substitution), and "ki" was added (1 insertion). That is 3 edits over 6 reference words, so WER is 0.50. Run the code and check.

Be careful what you count

Problem	Why it matters	What to do
Spelling variants	"subah" and "subha" may both be fine in romanised text, but WER counts an error	Agree a spelling guide for transcripts, or normalise both sides the same way
Script choice	"OTP" and "ओटीपी" mean the same, but are different strings	Decide the rule before you transcribe, and apply it to reference and output
Numbers	"299" and "two ninety nine" are both correct	Normalise numbers, or report them separately
Long words	In languages that join many parts into one long word, one small mistake makes the whole word wrong	Also report character error rate (CER), which counts character edits

Watch out

Normalise the reference and the system output with exactly the same rules, and write the rules down. If you clean only the output, or change rules between two systems, the comparison is not fair.

Testing fairly

1
Collect real audio
With consent, from the people and places your app will serve: phones, noise, different regions, genders and ages.
2
Transcribe carefully
Use trained native speakers and a written spelling guide. Have a second person check a sample.
3
Report by group
WER per language, dialect, noise level and speaker group, not only one average.
4
Look at the errors
Read the actual mistakes. Are they names, numbers, English words, or a whole dialect?
5
Fix and re-test
Improve the weakest group first, then test all groups again.

Note

Published WER numbers for a model are only true for the test set they were measured on. Clean read-aloud speech gives much lower WER than noisy, spontaneous speech. Always measure on audio like your users' audio.

Try this. Record yourself (only yourself) saying one sentence in a quiet room and near a fan or road. Use a free voice-typing tool on both. Write the reference and compute WER for each.

Watch and read more

AI4BharatIIT Madras · reading

Lab

WER and CER results for five sample transcripts, with and without normalisation, and a short error analysis.

~35 min

The problem

The starter has five pairs of reference and ASR output, in romanised Hinglish, English and Hindi.

Before running the code, work out the raw WER of pairs 2 and 4 by hand.
Run the code. Did your hand results match?
Compare raw WER with normalised WER. Which pairs changed, and why?
Look at pair 3. Is "299" versus "two ninety nine" really an error? Write a rule for numbers.
Add two pairs of your own in a language you know. Write the reference yourself; you may make up a realistic mistake for the output.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

import unicodedata

def edit_distance(ref: list, hyp: list) -> int:
    """Smallest number of substitutions, deletions and insertions."""
    prev = list(range(len(hyp) + 1))
    for i in range(1, len(ref) + 1):
        cur = [i] + [0] * len(hyp)
        for j in range(1, len(hyp) + 1):
            cost = 0 if ref[i - 1] == hyp[j - 1] else 1
            cur[j] = min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + cost)
        prev = cur
    return prev[-1]

def wer(ref: str, hyp: str) -> float:
    return edit_distance(ref.split(), hyp.split()) / len(ref.split())

def cer(ref: str, hyp: str) -> float:
    r, h = ref.replace(" ", ""), hyp.replace(" ", "")
    return edit_distance(list(r), list(h)) / len(r)

def normalise(text: str) -> str:
    text = unicodedata.normalize("NFC", text).lower()
    # Drop punctuation (P*) and symbols (S*) by Unicode category. Do not use
    # re's \w here: in Python it misses some Indic vowel signs.
    kept = "".join(" " if unicodedata.category(c)[0] in "PS" else c for c in text)
    return " ".join(kept.split())

# (what the person said, typed by a careful human) , (what the ASR system wrote)
samples = [
    ("mera PNR status batao", "mera pnr status batao"),
    ("kal subah das baje train hai", "kal subha das bje ki train hai"),
    ("I want to recharge my phone for 299", "i want to recharge my phone for two ninety nine"),
    ("मुझे कल बैंक जाना है", "मुझे कल बैंक जाना"),
    ("please send the OTP again", "please send the OTP again"),
]

for ref, hyp in samples:
    print(f"raw WER={wer(ref, hyp):.2f}  normalised WER={wer(normalise(ref), normalise(hyp)):.2f}"
          f"  CER={cer(normalise(ref), normalise(hyp)):.2f}  | {ref}")

# Corpus WER: total edits / total reference words (not the average of sentence WERs)
edits = sum(edit_distance(normalise(r).split(), normalise(h).split()) for r, h in samples)
words = sum(len(normalise(r).split()) for r, _ in samples)
print(f"corpus WER (normalised) = {edits}/{words} = {edits / words:.2f}")

Output

No output. Add print(...) to see values.

You are done when

Hand results for pairs 2 and 4 are shown with S, D and I counted
The write-up explains why pair 1 changes after normalisation (only the case differs)
There is a clear written rule for numbers, applied to both sides
Two new pairs were added and their results explained

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why does the code compute corpus WER as total edits divided by total words, instead of averaging the sentence WERs?Reveal

Averaging sentence WERs gives a short sentence the same weight as a long one. One error in a two-word sentence is a WER of 0.5, which would pull the average up a lot. Total edits over total reference words weights every word equally, which is the usual way to report WER for a test set.

Q2The comment says not to use re's \w for Indic text. What would go wrong?Reveal

In Python's re module, \w does not match some combining vowel signs used in Indic scripts. A pattern like \w+ would then cut words like "मुझे" into broken pieces, and WER would be computed on fragments. Removing punctuation by Unicode category keeps the words whole. Always test text-cleaning code on real Indic text.

Please sign in to continue.

Questions people ask

What is a good WER?

It depends on the task and the audio. A search box can survive some errors; a medical note or a bank transfer cannot. Set a target from what your users need, measure on realistic audio, and look at which errors matter most.

What is the difference between WER and CER?

WER counts word edits. CER counts character edits divided by the number of reference characters. CER is often useful for languages that join many parts into long words, or where word boundaries and spellings vary.

Why do names and numbers need special tests?

They carry the most important information in many tasks: who, where and how much. A small WER can hide a wrong phone number or a wrong village name. Test them with their own list and check them one by one.

Can I use recordings from YouTube to test?

Only if the licence and the platform's terms allow it, and even then think about privacy. It is better to use open datasets with clear licences, or to record volunteers who have given informed consent.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Speech recognition for Indian languages

Speech recognition for Indian languages

What makes it hard in India

Word error rate (WER)

Be careful what you count

Testing fairly

Collect real audio

Transcribe carefully

Report by group

Look at the errors

Fix and re-test

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask