Datasets and open projects for Indian languages
You do not have to start from zero. Learn about open projects such as AI4Bharat and Bhashini, where to find datasets, and why you must always check the licence.
In 60 seconds
Datasets and open projects for Indian languages
You do not have to start from zero. Learn about open projects such as AI4Bharat and Bhashini, where to find datasets, and why you must always check the licence.
Some open projects to know
| Project | What it is | What you may find |
|---|---|---|
| AI4Bharat | A research initiative at IIT Madras working on open AI for Indian languages | Open datasets, models and tools for text, translation, transliteration and speech |
| IndicTrans2 | An open machine translation model from AI4Bharat | Translation between English and Indian languages, and between Indian languages. Check its model card for the language list and licence. |
| Bhashini | India's national language translation mission, run by the Government of India | Language services and resources for building apps in Indian languages |
| Hugging Face | A popular platform that hosts datasets and models | Many community and research datasets for Indian languages, with dataset cards |
Finding a dataset on Hugging Face
Read before you use: a checklist
- Licence. Can you use it commercially? Must you give credit? Must you share changes under the same licence? Some datasets are for research only.
- Source. Where did the text or audio come from? Was it scraped from the web, translated, or recorded with consent?
- Consent and privacy. Does it contain personal data, names, phone numbers or voices of people who did not agree?
- Coverage. Which dialects, regions, genders and age groups are in it? Which are missing?
- Quality. Open a sample. Is the script right? Are there broken characters, machine-translated sentences or duplicates?
- Test overlap. Is any of it also in the test set you plan to use? If yes, your scores will look too good.
A quick look at a dataset in Python
datasets package from Hugging Face can load many public datasets. Replace the name with a dataset you found and whose licence you checked. Some datasets need you to accept terms on the website first.# pip install datasets
from datasets import load_dataset
NAME = "your-org/your-dataset" # put the dataset you chose here
ds = load_dataset(NAME, split="train", streaming=True) # streaming: no full download
for i, row in enumerate(ds):
print(row) # read real rows: script, quality, personal data?
if i == 4:
break- 1
Search
Find two or three candidate datasets for your language and task. - 2
Read the card
Licence, source, consent, coverage. - 3
Sample by eye
Read at least 50 rows. Ask a native speaker to read some too. - 4
Record it
Write down the dataset name, version, licence and date you downloaded it. - 5
Give back
Report errors to the owners, credit them, and share improvements if you can.
Watch and read more
Lab
A one-page dataset review of two datasets for one Indian language and one task.
The problem
Choose one language and one task (for example: speech recognition in Marathi, or translation between English and Assamese).
- Find two public datasets for it, for example on Hugging Face or from an open project.
- For each, fill in: name, owner, licence, commercial use allowed?, source of the data, consent described?, dialects or regions covered, size as stated on the card.
- Read at least 20 rows or listen to at least 5 clips. Note any quality problems.
- Decide which one you would use, for which purpose, and why. Say what is missing.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1A dataset card says the text was "collected from the public web". Does that make it safe to use for anything?Reveal
No. Public does not mean free of copyright, and web text can contain personal data such as names and phone numbers. You still need to check the licence, think about privacy, and consider filtering or removing personal data.
Q2Why keep a record of the dataset version and download date?Reveal
Datasets change: rows get fixed, removed or added, and licences can change. Without the version you cannot reproduce your results, explain what your model learned from, or respond if someone asks you to remove their data.
Questions people ask
Can I train a commercial model on any dataset I find online?
No. Each dataset has its own licence and some forbid commercial use. Web text can also include copyrighted or personal material. Read the licence, keep a record, and ask a legal expert for important products.
What is a model card or dataset card?
A short document that comes with a model or dataset. It explains what it is for, how it was made, what it covers, its limits and its licence. Good cards make honest use much easier.
Is machine-translated training data a problem?
It can be. Text translated by machine may sound unnatural, repeat the translator's mistakes and miss how people really speak. It can still help, but mark it clearly and do not use it as your only test data.
My language has almost no datasets. What now?
Look for community groups, universities and language institutions working on it. Small, careful, consented datasets are valuable. The last lesson of this track covers how to collect data ethically.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.