AI on phones and edge devices
Running AI on the phone or a small device keeps data local and works offline. Learn the runtimes, the battery and memory limits, and how to update models safely.
In 60 seconds
AI on phones and edge devices
Running AI on the phone or a small device keeps data local and works offline. Learn the runtimes, the battery and memory limits, and how to update models safely.
Why do it on the device?
- Privacy: photos, voice and messages can stay on the device.
- Offline: works in areas with weak or no signal, which still matters in many parts of India.
- Speed: no network round trip for small, quick tasks like keyboard suggestions or live captions.
- Cost: the user's device does the work, so you pay less for servers.
On-device runtimes (in general terms)
| Kind of runtime | Examples (names and features change; check current docs) |
|---|---|
| Cross-platform | ONNX Runtime; llama.cpp, which can also be built for phones |
| Platform-specific | Core ML on Apple devices; Google's LiteRT (formerly TensorFlow Lite) |
| Built into the operating system | Some phones now include AI models that apps can call through system features |
The limits you must design for
| Limit | What happens | What helps |
|---|---|---|
| Memory | The app is closed by the system if it uses too much | Small, quantised models; load only when needed |
| Battery | Heavy use drains the battery and users uninstall the app | Short tasks, smaller models, using the NPU where possible |
| Heat | The phone slows itself down to cool off | Avoid long continuous generation |
| Storage and download size | Users on limited data may not download a large model | Download the model separately, over Wi-Fi, with user consent |
| Device variety | A model that runs on a new flagship may fail on an older budget phone | Test on low-end phones that your users actually own |
Updating models on devices
- 1
Version every model
Give each model file a version number and a checksum. - 2
Download safely
Use HTTPS, check the checksum or signature before loading, and download over Wi-Fi by default. - 3
Roll out slowly
Send the new model to a small share of users first and watch for problems. - 4
Keep a fallback
If the new model fails to load or performs badly, go back to the previous one. - 5
Respect data limits
Tell users the size, and let them choose when to download.
- Bundling a large model inside the app so every install is huge
- Testing only on the newest phone
- Loading any model file the server sends, with no checks
- Offering the model as an optional download with the size shown
- Testing on the low-cost phones your users own
- Checking a signature or checksum before loading a model
Watch and read more
Lab
An on-device design plan for one feature, with limits, testing devices and an update process.
The problem
Feature: "an app for ASHA workers that turns a spoken note in Hindi into a short written summary, working offline, with notes never leaving the phone."
- Choose what runs on the device (for example speech recognition and summarising) and whether anything ever goes to a server.
- Estimate the model size you can afford, using the memory rule from lesson 3, for a phone with limited RAM.
- List three test phones (by price range and memory, not brand) and what you will measure on each: speed, battery use, heat, crashes.
- Write the model update process in five steps, including a fallback.
- Write two sentences for the privacy notice that are true for your design.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Why might a model run faster on the phone's NPU in one app, but slower in another app with the same model?Reveal
NPUs support only certain operations and number formats. If the runtime or the model's conversion uses an operation the NPU does not support, that part falls back to the CPU or GPU, and moving data between chips costs time. Different runtimes, conversions or quantisation settings can change which parts run where. You have to measure on the real device.
Q2Your on-device model gives wrong medical-sounding advice in some cases. Is on-device privacy a good enough reason to keep it?Reveal
No. Privacy does not excuse harmful answers. Limit the feature to what it does reliably (for example summarising, not advising), show clear warnings, keep a human health worker in charge of decisions, and improve or replace the model. Advice about health should come from trained professionals.
Questions people ask
Can a phone really run a language model?
Small, quantised language models can run on many recent phones, though more slowly than on a server. Smaller models for tasks like speech recognition, translation of short text, photo tagging or text suggestions have run on phones for years.
What is an NPU?
A neural processing unit: a part of the chip designed for the maths that AI models use. It can run some models faster and with less battery than the CPU. Which models and formats it supports depends on the phone and the runtime.
Is it better to do everything on the device?
Not always. A common design is hybrid: simple or private tasks run on the device, and harder tasks go to a server when there is a connection and the user agrees.
How do I protect the model file on the device?
Anyone with the device can usually get at the file, so do not rely on keeping it secret. Protect users instead: check signatures before loading updates, and never put secrets such as API keys inside the model or the app.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.