Tokens, scripts and the cost of Indic text
Tokenizers trained mostly on English often split Indian-language text into more pieces. That means higher cost and less room in the context window. Measure it yourself.
In 60 seconds
Tokens, scripts and the cost of Indic text
Tokenizers trained mostly on English often split Indian-language text into more pieces. That means higher cost and less room in the context window. Measure it yourself.
Three ways to count the same sentence
| Unit | What it counts | Why it matters |
|---|---|---|
| Characters (code points) | Unicode code points, the numbers the computer stores for each letter or sign | A visible letter in an Indic script is often more than one code point |
| UTF-8 bytes | The bytes used to store or send the text | Basic Latin letters take 1 byte. Letters in scripts such as Devanagari, Bengali, Tamil or Telugu take 3 bytes each. Arabic-script letters, used for Urdu, take 2. |
| Tokens | The pieces the model reads, which depend on the tokenizer | You pay per token, and the context window is measured in tokens |
See it in Python
import unicodedata
samples = {
"English": "Where is the railway station?",
"Hindi": "रेलवे स्टेशन कहाँ है?",
}
for name, text in samples.items():
chars = len(text) # Unicode code points
utf8 = len(text.encode("utf-8")) # bytes on disk / on the wire
print(f"{name:8} chars={chars:3} utf8_bytes={utf8:3} bytes_per_char={utf8 / chars:.2f}")
# Look inside one Hindi syllable: one visible unit can be several code points
for ch in "स्टे":
print(f"U+{ord(ch):04X} {len(ch.encode('utf-8'))} bytes {unicodedata.name(ch)}")Why more tokens hurts
- Higher cost. Hosted models usually charge per token. More tokens for the same meaning means a bigger bill for users of that language.
- Less context. The context window is counted in tokens. If a document needs more tokens, less of it fits, so the model "remembers" less.
- Slower answers. Models produce output one token at a time, so more output tokens take longer.
- Worse learning. When words are cut into meaningless pieces, the model may learn their meaning less well.
What builders can do
- 1
Measure first
Count tokens for real user text in each language with your actual tokenizer. - 2
Compare models
Tokenizers differ. Some newer ones are trained with more multilingual data. Test several. - 3
Budget per language
Set token limits and prices with the real token counts, not English ones. - 4
Trim the prompt
Keep long fixed instructions short. Some teams write system instructions in English and ask for answers in the user's language; test whether that keeps quality. - 5
Consider Indic-trained models
Models and tokenizers trained with more Indian-language data may be more efficient. Check their licences and quality.
- "A token is about four characters" for every language
- Setting the same length limit for all languages
- Testing costs only with English text
- Counting tokens with the real tokenizer on real text
- Setting limits based on each language's token counts
- Tracking cost per language in your dashboard
Watch and read more
Lab
A small table of characters, bytes and tokens for the same sentence in at least three languages.
The problem
Run the starter code. Then:
- Add at least two more sentences with the same meaning in languages you know or can check with a speaker. Type them yourself or ask a friend who speaks the language.
- If you can install it,
pip install tiktokento see token counts from one public tokenizer. If not, the code still shows characters and bytes. - Make a table: language, characters, bytes, tokens, and tokens compared with English.
- Write two sentences on what this would mean for a chatbot's cost and context in each language.
Do not state results you did not measure. If you only measured bytes, say so.
# Same meaning, different languages. Replace or add your own lines.
sentences = {
"English": "Please send the bill on WhatsApp.",
"Hindi": "कृपया बिल व्हाट्सऐप पर भेजिए।",
# "Tamil": "...", # add a language you know, typed by you
}
try:
import tiktoken # optional: pip install tiktoken
enc = tiktoken.get_encoding("cl100k_base")
except ImportError:
enc = None
base = None
for name, text in sentences.items():
chars = len(text)
utf8 = len(text.encode("utf-8"))
tokens = len(enc.encode(text)) if enc else None
if base is None and tokens:
base = tokens
ratio = f" x{tokens / base:.1f} vs first line" if tokens and base else ""
print(f"{name:8} chars={chars:3} bytes={utf8:3} tokens={tokens}{ratio}")You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Two tokenizers give very different counts for your Kannada text. Which should you trust?Reveal
Both are correct for their own model. Token counts belong to a tokenizer, not to a language. What matters is the tokenizer of the model you will actually use. If you are choosing a model, compare cost and quality together: a model with fewer tokens but worse answers is not a win.
Q2Could you fix the cost gap by transliterating Indic text into Roman letters before sending it?Reveal
It may reduce tokens with some tokenizers, but it is risky. Transliteration has no single standard, it can lose meaning and sounds, the model may understand it less well, and the user usually wants the answer in their own script. Test quality carefully before doing this, and never do it silently to text the user will see.
Questions people ask
Is UTF-8 unfair to Indian scripts?
UTF-8 is a storage format. It uses more bytes for letters outside the basic Latin range, and that is a fixed design choice that works for every language. The bigger issue is the tokenizer: what it learned to treat as common pieces decides how many tokens a sentence becomes.
Why is a "character" count confusing for Indic text?
Python's len() counts Unicode code points. A letter you see on screen, like a consonant with a vowel sign, can be two or more code points. What a reader thinks of as one letter is called a grapheme cluster. The third-party regex package can count those with \X.
Can I count tokens without paying?
Yes, usually. Many open tokenizers can be run locally, for example through Hugging Face libraries, and some providers publish their tokenizers or offer a free token-counting tool. Check the current docs for the model you use.
Should I write prompts in English to save tokens?
Sometimes it helps, but test it. The instructions may be cheaper in English, but the answer should still be in the user's language and natural. Always check quality in the target language, not only cost.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.