Building with multimodal APIs
How to send images and audio to a model API, what it costs, how to protect people's privacy in photos, and how to use it well for accessibility, like writing alt text.
In 60 seconds
Building with multimodal APIs
How to send images and audio to a model API, what it costs, how to protect people's privacy in photos, and how to use it well for accessibility, like writing alt text.
How the image travels
- Inline: the file's bytes are encoded as text using base64 and put inside the request. Base64 makes the data about a third larger.
- By link or upload: some providers accept a URL or a file you uploaded earlier. Make sure the link is not something private you would not want the provider to fetch.
- Field names, size limits and supported formats differ by provider and change over time. Always check the current docs.
import base64
import pathlib
def call_model(parts: list) -> str:
... # plug in your provider's SDK or HTTP call here
raise NotImplementedError
def image_part(path: str, media_type: str = "image/jpeg") -> dict:
data = pathlib.Path(path).read_bytes()
return {
"type": "image",
"media_type": media_type,
"data": base64.b64encode(data).decode("ascii"),
}
def describe_for_alt_text(path: str) -> str:
parts = [
image_part(path),
{"type": "text", "text": (
"Write alt text for this image for a screen reader user. "
"One or two sentences, under 125 characters if possible. "
"Describe what matters for understanding the page. "
"Do not guess names, ages, caste, religion or health of people. "
"If there is important text in the image, include it exactly."
)},
]
return call_model(parts)Cost: images are many tokens
- 1
Resize first
Shrink images to the smallest size that still shows what you need. - 2
Crop
Send only the part that matters, such as the address block of a form. - 3
Avoid resending
In a long chat, the same image may be counted again each turn. Summarise it once, then refer to the summary where possible. - 4
Set limits
Cap file size, images per request and requests per user. - 5
Measure
Log token counts per request and watch the bill.
Privacy: photos say more than you think
| What a photo may contain | What to do |
|---|---|
| Faces of people who did not agree | Get consent, or blur faces you do not need |
| Location, device and time in hidden metadata (EXIF) | Strip metadata before sending or storing |
| Documents: Aadhaar, PAN, bank details, medical reports | Mask numbers you do not need; avoid sending if you can |
| Things in the background: a screen, a letter, a number plate | Crop to what matters |
Accessibility: a strong, good use
- "Image of a picture"
- "A happy young Punjabi woman" (guessing identity and feelings)
- A 200-word description of a decorative border
- "Bar chart: monsoon rainfall rose each year from 2021 to 2024" (with the true values checked)
- "A woman holds a certificate on stage"
- Empty alt text for purely decorative images, so screen readers skip them
Watch and read more
Lab
Alt text for five images, drafted with AI and corrected by you, with notes on every fix.
The problem
Use any free chatbot that accepts images, or the starter code with your own provider. Use images you have the right to use, with no private documents.
- Choose five images from a school, college or community website or a presentation you made: a photo with people, a chart or graph, an image with text (a poster), a product photo, and a purely decorative image.
- Ask the model to write alt text, using the instructions in the starter code.
- For each image, check the draft: are facts, numbers and text correct? Does it guess identity, age, religion or feelings? Is it too long? Is the decorative image correctly given empty alt text?
- Write your corrected version next to the AI draft and a short note on what you changed and why.
- Optional: resize one image to half its width and compare the answer. Did anything change?
import base64
import pathlib
def call_model(parts: list) -> str:
... # plug in your provider's SDK or HTTP call here
raise NotImplementedError
PROMPT = (
"Write alt text for this image for a screen reader user. "
"One or two sentences, under 125 characters if possible. "
"Describe what matters for understanding the page. "
"Do not guess names, ages, caste, religion or health of people. "
"If there is important text in the image, include it exactly. "
"If the image is purely decorative, reply with exactly: DECORATIVE"
)
MEDIA = {".jpg": "image/jpeg", ".jpeg": "image/jpeg",
".png": "image/png", ".webp": "image/webp"}
MAX_BYTES = 2_000_000 # your own limit; check your provider's docs
def image_part(path: pathlib.Path) -> dict:
data = path.read_bytes()
if len(data) > MAX_BYTES:
raise ValueError(f"{path.name} is too large; resize it first")
encoded = base64.b64encode(data).decode("ascii")
print(f"{path.name}: {len(data)} bytes, {len(encoded)} as base64")
return {"type": "image", "media_type": MEDIA[path.suffix.lower()],
"data": encoded}
def draft_alt_text(path: pathlib.Path) -> str:
reply = call_model([image_part(path), {"type": "text", "text": PROMPT}])
return "" if reply.strip() == "DECORATIVE" else reply.strip()
if __name__ == "__main__":
for p in sorted(pathlib.Path("images").iterdir()):
if p.suffix.lower() in MEDIA:
try:
print(p.name, "->", repr(draft_alt_text(p)))
except NotImplementedError:
print(p.name, "-> (plug in call_model first)")You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your app lets users upload photos for AI descriptions. How do you stop costs from running away?Reveal
Resize and compress images on your side before sending, and reject files above a size limit. Cap images per request and requests per user per day. Cache results for the same image so it is not sent twice. Log token counts per request and set alerts on the daily bill. Use a smaller, cheaper model for simple jobs and a larger one only when needed.
Q2Why is "wrong alt text is worse than none" true for many screen reader users?Reveal
A sighted user can see that a caption is wrong. A blind user trusts the alt text because it is their only access to the image. If it states a wrong number in a chart or a wrong instruction on a poster, they may act on it without any way to notice. Missing alt text at least signals that information is missing, so they can ask someone. That is why a human check matters most for images people will act on.
Questions people ask
Should I send the whole photo or a crop?
A crop, whenever you can. It is cheaper, more private and often more accurate, because the model sees the important part in more detail.
Can I send audio or video the same way?
Some providers accept audio and video, others do not, and limits differ. Some want you to upload the file first. Check the current docs, and remember that audio and video can be expensive and contain even more personal data.
Is AI alt text good enough to publish without review?
For decorative or low-stakes images, a quick check may be enough. For charts, instructions, products, news photos and anything with numbers or text, a person should review it. Wrong alt text is worse than none, because the user cannot see that it is wrong.
How do I strip location data from photos?
Many phones have a setting to not add location to photos, or to remove it when sharing. Image libraries and tools can also save a copy without metadata. After stripping, check the file with an EXIF viewer to confirm.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.