Speech models: listening and talking
Speech recognition turns voice into text, text-to-speech turns text into voice, and newer assistants talk directly. Learn how each works, why speed matters, and why voices need consent.
In 60 seconds
Speech models: listening and talking
Speech recognition turns voice into text, text-to-speech turns text into voice, and newer assistants talk directly. Learn how each works, why speed matters, and why voices need…
Listening: automatic speech recognition
- Accents, background noise (a busy railway platform, a fan, traffic) and phone-quality audio all make it harder.
- Code-mixing, like switching between Hindi and English in one sentence, is common in India and still challenging for many systems.
- Names, place names and technical words are frequent errors. Some tools let you give a list of expected words.
Talking: text-to-speech
Two ways to build a voice assistant
| Pipeline (cascade) | Speech-to-speech | |
|---|---|---|
| How it works | ASR, then a text language model, then TTS | One model takes in audio and produces audio, often with text inside as well |
| Strengths | Easy to inspect: you can log and check the text at each step; parts can be swapped | Can hear tone and emotion, can respond faster and more naturally |
| Weaknesses | Adds delay at each step; loses tone of voice | Harder to inspect and control; may be costlier |
Why latency matters
- 1
Streaming
Transcribe while the person is still speaking, not after. - 2
Turn detection
Guess when the person has finished, without cutting them off mid-thought. - 3
Start speaking early
Begin playing the first words of the answer while the rest is still being made. - 4
Barge-in
Stop talking at once if the person interrupts.
Voices need consent
- Cloning a friend's voice "for a joke" without asking
- Using a celebrity's voice in an advert
- Recording customers to train a voice model without telling them
- Written, informed consent from the voice owner, for a stated use
- A clear label that the voice is synthetic
- A way for people to withdraw consent and have data deleted
Watch and read more
Lab
A small latency and accuracy log for a voice assistant, plus a one-page voice consent note.
The problem
Use any voice assistant or voice mode on your phone or computer.
- Ask five short questions, including one in your home language and one mixing two languages. Use a stopwatch (a friend can help) to time from the end of your question to the first word of the reply.
- For each, write: the time, whether it understood you, and whether it cut you off or interrupted.
- Interrupt it once while it is talking. Does it stop?
- Now imagine your college wants to make a synthetic voice of a retired teacher for an audio library. Write a short consent note (under 150 words) that says what the voice will be used for, what it will not be used for, how it will be labelled, and how the teacher can withdraw.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Why is turn detection hard, and what is the trade-off?Reveal
People pause mid-sentence to think, and a pause can look the same as the end of a turn. If the system waits longer to be sure, every reply feels slow. If it jumps in quickly, it interrupts people. Good systems combine silence length with signs from the words themselves (does the sentence sound finished?), and let users interrupt easily when it gets it wrong.
Q2A bank wants a speech-to-speech assistant for customers. What would you require before launch?Reveal
A way to record and review what was said and done, which may mean also producing text. Strict limits on actions: no money moves on voice alone. Testing across languages, accents and noisy settings with real users. Clear disclosure that it is an AI. Short, stated retention for recordings, and no use for training without consent. An easy way to reach a human.
Questions people ask
Why do voice assistants sometimes cut me off?
They are guessing when you have finished speaking, usually from a short silence. A pause to think can look like the end of your turn. Better systems use the meaning of what you said too, but it is still a hard problem.
Is speech-to-speech always better than a pipeline?
No. Pipelines are easier to inspect, test and control, which matters in banking, health or government services. Speech-to-speech can feel more natural. Many products choose based on the job.
Can ASR be used for Indian languages?
Yes, and it has improved a lot, with open work from groups such as AI4Bharat at IIT Madras and the government's Bhashini mission. Quality still varies by language, accent and topic, so test with real speakers.
Is my voice personal data?
Often, yes. A voice can identify you, and recordings can reveal health, emotion and location. Many privacy laws, including India's Digital Personal Data Protection Act, treat such data with care. Ask how long recordings are kept and whether they are used for training.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.