Machine translation and how to judge it
Machine translation is useful and imperfect. Learn to judge it with people, with back-translation and with metrics like BLEU and chrF, and know the limits of each.
In 60 seconds
Machine translation and how to judge it
Machine translation is useful and imperfect. Learn to judge it with people, with back-translation and with metrics like BLEU and chrF, and know the limits of each.
What can go wrong
- Meaning errors. A negation is lost ("do not" becomes "do"), or a number changes.
- Domain terms. A legal, medical or farming term is translated as an everyday word with a different meaning.
- Names. A person's name or a place is translated as if it were a common word.
- Register and tone. Too formal, too casual, or the wrong form of "you" for the reader.
- Fluent nonsense. A smooth sentence that says something the source did not say.
Three ways to judge a translation
| Method | How it works | Strengths | Limits |
|---|---|---|---|
| Human evaluation | Bilingual reviewers rate or correct translations using a rubric | The best judge of meaning, tone and domain terms | Slower and costs money; reviewers need clear guidelines and can disagree |
| Back-translation check | Translate the output back to the source language and compare | Cheap way to spot big meaning errors when you do not read the target language | Both directions can share the same mistake; a good back-translation does not prove the output is natural |
| Automatic metrics | Compare the output with one or more human reference translations | Fast and repeatable; good for comparing versions of a system | Needs good references; can reward word overlap even when meaning is wrong |
BLEU and chrF, in plain words
sacrebleu Python package computes both in a standard way.A simple human rubric
| Score | Adequacy (meaning) | Fluency (natural language) | Terminology |
|---|---|---|---|
| 4 | All meaning kept | Reads like a native speaker wrote it | All key terms correct |
| 3 | Small details lost | Understandable, a little unnatural | One minor term issue |
| 2 | Important meaning lost or added | Hard to read in places | A key term wrong |
| 1 | Wrong or dangerous meaning | Not understandable | Several key terms wrong |
An example: how OmniLingo handles translation
- 1
Translate once, cache
Each translated piece is stored, so the same text is not translated and paid for again. - 2
Human review
People can correct a translation. Corrected pieces are marked as reviewed. - 3
Protect reviewed work
Reviewed translations are never overwritten by a new machine translation. - 4
Fail honestly
If machine translation is not available, the course shows the English original rather than an invented translation. - 5
Keep code as code
Code samples are not translated, because translating code would break it.
Watch and read more
Lab
Ten translations scored by two reviewers with a rubric, an agreement check, and a list of error types.
The problem
Work with a friend who also reads the target language.
- Pick 10 English sentences from real life: a bank SMS, a train notice, a school circular, a medicine label (do not act on it), a farming tip. Include at least 3 with numbers or names.
- Machine-translate them into an Indian language you both read well.
- Each of you scores every translation 1 to 4 for adequacy, fluency and terminology, using the rubric in the lesson. Do not look at each other's scores.
- Put your scores into the starter code and run it.
- Discuss every item the code flags. Write the error type (meaning, name, number, term, tone) and a corrected translation.
Practice lab
Edit the code and run it. Python runs in your browser — nothing is sent to our server.
No output. Add print(...) to see values.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your two reviewers agree on fluency but often disagree on adequacy. What does that suggest, and what would you do?Reveal
They may understand "meaning kept" differently, for example whether a small lost detail is a 3 or a 2. Write clearer rules with examples for each score, have them score a few practice items together, then score new items independently again. If one reviewer does not know the domain, add a glossary or a domain expert.
Q2A new translation model raises chrF on your test set, but your support team gets more complaints. How can you find out what happened?Reveal
Look at the complaints and find the sentences involved. Check whether your test set looks like real user text, whether the model now gets names, numbers or negation wrong while matching more words, and whether the test set leaked into training. Add the failing real sentences to your test set and run a human review on them before and after the change.
Questions people ask
Is back-translation enough to check my app's translations?
No. It is a quick screen for big meaning errors. It cannot tell you if the output sounds natural, and errors can cancel out on the way back. For anything important, use bilingual human reviewers.
Which metric should I report, BLEU or chrF?
Many teams report more than one, computed with a standard tool so others can compare. chrF is often preferred for languages with rich word forms. Whatever you report, add human evaluation on a sample and say which test set you used.
Can an LLM judge translation quality?
It can help to flag likely problems at scale, but it can be wrong, especially in languages it saw less of. Check its judgements against bilingual human reviewers before you trust it, and never use it as the only check for important text.
What should never be machine-translated without review?
Anything where a mistake can cause harm: medical instructions, legal notices, safety warnings, financial terms, and official forms. Have a qualified human review these.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.