What multimodal means
Text is only one way to hold information. Learn how images, audio and video are turned into tokens and embeddings, so one model can work with all of them.
In 60 seconds
What multimodal means
Text is only one way to hold information. Learn how images, audio and video are turned into tokens and embeddings, so one model can work with all of them.
Everything becomes numbers
| Modality | What the raw data is | How it is often cut into pieces | Then |
|---|---|---|---|
| Text | Letters and words | Tokens: words or parts of words | Each token becomes an embedding |
| Image | A grid of pixels, each with red, green and blue values | Small square patches, for example 16 by 16 pixels | Each patch becomes an embedding |
| Audio | A wave: thousands of loudness values every second | Short slices of time, often turned into a picture of pitch over time (a spectrogram) | Each slice becomes an embedding |
| Video | Many images per second, often with sound | Patches in space and in time, or a few chosen frames | Each piece becomes an embedding |
Encoders: translators into the model's language
- 1
Raw input
A photo of a railway ticket, plus the question "Which platform?" - 2
Split
The photo is cut into patches. The question is cut into text tokens. - 3
Encode
An image encoder turns patches into embeddings. Text tokens get their own embeddings. - 4
Mix
Both sets of embeddings go into one sequence. Attention lets text tokens "look at" image patches. - 5
Answer
The model writes the answer as text, one token at a time.
A shared space of meaning
- The model "looks" at a photo the way you do
- Audio is first always turned into text
- More modalities means the model is smarter
- The model receives numbers for small patches and learns patterns across them
- Some systems transcribe first; others work on sound directly
- Each modality adds new ways to fail, and needs its own testing
Why this matters to you
- Cost: images and audio usually become many tokens. A single photo can cost more than a long paragraph. Check your provider's current docs.
- Errors: each step, like shrinking an image, can lose detail. Small print on a medicine strip may simply not survive.
- Privacy: a photo can hold far more than you meant to share: faces, addresses, number plates, and hidden location data.
Watch and read more
Lab
A hand-drawn map of how one real message becomes pieces a model can read.
The problem
Pick one everyday message that mixes modes, for example a WhatsApp message with a photo of a crop leaf and a voice note in your language asking "what is wrong with my plant?"
- List each modality in the message.
- For each, write how it would likely be cut into pieces (tokens, patches, time slices) and roughly how many pieces you guess. Label your numbers as guesses.
- For each, write one detail that could be lost on the way (for example, a small spot on the leaf after the image is shrunk).
- Draw a simple flow like the one in this lesson, from raw input to answer.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1If images and text end up as embeddings in one sequence, why do vision models still make mistakes that a text model would not?Reveal
Because the conversion is lossy and learned. The image may be shrunk, patches may cut through small details, and the encoder was trained to capture what captions usually mention, not every fact in the picture. The language model then reasons over this summary. If a detail never made it into the embeddings, no amount of reasoning can recover it.
Q2Why is it hard to train good multimodal models for many Indian languages?Reveal
The lining-up of meaning is learned from paired data, such as images with captions or speech with transcripts. There is much less of this paired data for many Indian languages than for English. So a model may understand a photo well when asked in English and less well when asked in Odia or Santali. Open projects that collect such data with consent help close the gap.
Questions people ask
Is a multimodal model one model or several joined together?
It can be either. Many systems join a separate image or audio encoder to a language model. Others are trained as one model from the start. From outside, they can look the same; the design matters for cost, speed and the kinds of mistakes they make.
Does the model see the whole image in full detail?
Often not. Many systems resize the image first, so very small text or far-away objects can be lost. Some split large images into tiles to keep detail. If detail matters, crop to the important part before sending.
What is an embedding, again?
A list of numbers that stands for a piece of data. Pieces with similar meaning get similar lists. You can think of each embedding as a point on a very large map, where similar things sit near each other.
Can one model also create images, not just read them?
Some can. Reading a modality and producing it are different skills. Later lessons cover how images, video and sound are generated, and models that do several of these at once.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.