Multimodal and generative AIMiddleLesson 2436 min read

How vision-language models see

Image patches, a vision encoder, and CLIP-style matching of pictures and words. Plus the mistakes these models still make with counting, small text and left-right.

✦ Ask the tutor🧪 Lab · 30 min📝 Lesson test
Lesson in motion

In 60 seconds

How vision-language models see

Image patches, a vision encoder, and CLIP-style matching of pictures and words. Plus the mistakes these models still make with counting, small text and left-right.

1/5
In simple words
Imagine you cut a photo into a grid of small stamps and give them to a friend with their positions written on the back. Your friend can still guess "this is a cricket match", but might not be able to count every fielder. That is roughly how an AI "sees".
A vision-language model (VLM) takes in images and text and usually answers in text. You use one when you ask a chatbot to read a menu, explain a graph or describe a photo.

Step 1: patches and a vision encoder

The image is resized and cut into small square patches. Each patch becomes an embedding, and its position is recorded. A vision encoder, often a transformer, then lets patches share information through attention, so a patch showing part of a bat can "know" it sits next to a patch showing a hand.

Step 2: CLIP-style matching of pictures and words

How does the encoder learn what things mean? One well-known method is CLIP, from OpenAI. It trains two encoders together: one for images and one for text. It is shown a very large number of image-caption pairs from the internet.
  1. 1

    Take a batch

    Many images, each with its own caption.
  2. 2

    Encode both

    Every image and every caption becomes one embedding.
  3. 3

    Compare all pairs

    Measure how close each image is to each caption.
  4. 4

    Pull and push

    Pull matching pairs closer. Push mismatched pairs apart.
  5. 5

    Repeat

    After enough examples, pictures and words that mean the same thing sit near each other.
This is called contrastive learning. A useful side effect: you can classify an image with no extra training. Compare it with "a photo of a dog", "a photo of a cow" and "a photo of an auto-rickshaw", and pick the closest caption.

Step 3: joining the encoder to a language model

Many VLMs take a trained vision encoder, add a small adapter that converts its embeddings into the language model's "format", and then train the whole system on images with questions and answers. The language model now sees image patches as extra tokens in its input, and attends to them while writing.
Note
Designs vary. Some models are trained on images and text together from the start. The exact architecture of many commercial models is not public.

What they often get wrong

WeaknessExampleWhy it happens
Counting"How many people are on this bus roof?" gets a confident wrong numberPatches blur objects together; captions in training rarely count exactly
Small or dense textMisreads a medicine batch number or a price on a crowded shop boardResizing loses fine detail; text may fall across patch edges
Spatial reasoningSays the cup is left of the plate when it is on the rightCaptions rarely describe exact positions, so the skill is learned weakly
Rare thingsNames a local festival or a regional dish wronglyLess training data for them
Hallucinated detailsDescribes a clock that is not in the photo, because kitchens usually have oneThe model fills in what is typical, not what is there
Watch out
A VLM's answer sounds just as sure when it is wrong. Never rely on it alone for things that matter: medicine labels, legal documents, meter readings for a bill dispute, or anything about a person's identity.
Asking badly
  • "What does this say?" with a blurry full-page photo
  • Trusting a count of 40 objects
  • Asking it to identify a stranger in a photo
Asking well
  • Crop to the part you need, in good light, then ask
  • Ask it to list items one by one, then count yourself
  • Ask only about the content, never to identify people
Try this. Take a photo of a shelf with 12 to 20 items. Ask a VLM to count them. Then ask it to list them one by one. Which answer is closer?

Watch and read more

arXiv: search for CLIP and vision-language papersarXiv · reading

Lab

A small error report showing where a vision-language model was right and wrong on your own photos.

~30 min

The problem

Use any free chatbot that accepts images. Use only photos you took yourself, with no faces or private documents.

  1. Take five photos: (a) a group of 10 to 25 similar objects, such as coins or pens; (b) a label with small print, such as a food packet; (c) three objects in a row on a table; (d) a busy street or market scene; (e) something local and less common, such as a regional snack or tool.
  2. For each, ask one precise question: the count; the exact text of one small line; "which object is on the far left?"; "describe this scene"; "what is this called?".
  3. Write down the answer and the truth. Mark each as right, partly right or wrong.
  4. For each wrong answer, try one fix: crop, better light, ask it to list items one by one. Note whether it helped.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1You want to use a VLM to read electricity meter photos sent by customers. How would you make this reliable?Reveal

Do not trust one reading. Ask customers for a close, well-lit photo of the digits. Ask the model for the reading in a fixed format, and check it in code: right number of digits, not lower than last month, not wildly higher. Send doubtful cases to a person. Measure accuracy on a test set of real photos with known readings before launch, and keep measuring after.

Q2Why can CLIP-style matching carry social bias?Reveal

It learns from captions written by people on the internet. If those captions link certain jobs, clothes or skin tones with certain words more often, the embeddings learn the same links. The model can then match a photo of a person to a caption based on stereotypes. Testing with balanced examples and not using such models to judge people are important safeguards.

Please sign in to continue.

Questions people ask

Does CLIP generate captions?

No. CLIP scores how well an image and a piece of text match. It does not write text on its own. But its image encoder, or ones trained in a similar way, is often used inside systems that do.

Why are VLMs better at describing a scene than counting?

Their training data is mostly captions like "a crowded market in the evening", which describe the gist. Exact counts and positions are rarely written down, so the model learns them weakly.

Will these weaknesses go away?

Newer models have improved on several of them, and techniques like sending higher-resolution tiles help. But you should still test the model on your own images, because a model that is good on average can fail on your exact case.

Is it safe to send photos of people to a VLM?

Think first. The photo leaves your device and may be stored by the provider. Get consent from people in the photo where you can, blur faces you do not need, and never use these tools to identify strangers.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress