The short version: Run ML on the phone when you need instant response, offline use, privacy or lower per-call cost. In 2026 that means Core ML on iOS, LiteRT (formerly TensorFlow Lite) on Android, and the built-in Apple and Gemini Nano language models where devices support them. Quantise, then test accuracy.
Last updated: 21 October 2026.
On-device AI means the model runs on the user's phone instead of on your server. The photo, voice note or document never leaves the device, results come back without a network round trip, and you do not pay for each inference. The cost moves elsewhere: model size, device fragmentation and a harder update process.
This guide covers when that trade is worth it, which Apple and Google tools to use in 2026, and how to ship and update models without bloating your app.
When does on-device ML beat a cloud API?
On-device wins on latency, privacy, offline use and running cost. Cloud wins on model size, freshness and control. Apple's Core ML documentation states the core benefit plainly: running a model strictly on a person's device removes the need for a network connection, which helps keep their data private (Apple, Core ML).
| Factor | On-device | Cloud API |
|---|---|---|
| Latency | No network round trip; good for camera and live input | Depends on connection and server load |
| Privacy | Data stays on the phone | Data leaves the device; needs consent and controls |
| Offline | Works with no signal | Fails without a connection |
| Cost per call | No server inference cost | Paid per request or per GPU hour |
| Model size and quality | Limited by phone memory and battery | Can run the largest models |
| Updates | Ship a new model to every device | Change it once on the server |
| Device coverage | Varies by chip, RAM and OS version | Same for every user |
Our rule of thumb: if the feature works on live camera or audio frames, handles data users would hesitate to upload, or must work in a lift or a field, start on-device. If it needs a large model or current data, start in the cloud. Many apps end up hybrid, with a small on-device model for the common case and a server call for the hard one.
What is the current Apple stack for on-device ML?
Core ML is Apple's framework for running models in apps across iOS, iPadOS, macOS, watchOS, tvOS and visionOS. It schedules work across the CPU, GPU and Neural Engine while keeping memory use and power consumption down (Apple, Core ML). You train in PyTorch or another library, then convert with Core ML Tools, or train smaller models directly in Create ML.
For language tasks, Apple added the Foundation Models framework in iOS 26. It gives apps access to the on-device model behind Apple Intelligence for tasks such as summarisation, entity extraction and text understanding, with guided generation into Swift types and tool calling. It requires a device that supports Apple Intelligence (Apple, Foundation Models). That makes it a strong option for text features with no model to ship, and a feature you must hide or replace on older phones.
What is the current Google and Android stack?
TensorFlow Lite is now LiteRT. Google announced the rename on 4 September 2024 and said existing TensorFlow Lite apps are not affected (Google Developers Blog). Moving from TensorFlow Lite to LiteRT's classic Interpreter API needs only a package name change. One caveat: Google's migration guide says apps using the Swift and Objective-C iOS SDKs should stay on the TensorFlow Lite packages for now (Google AI Edge, LiteRT migration).
LiteRT 2.x adds the CompiledModel API, which Google recommends over the Interpreter API for hardware acceleration on GPUs and NPUs. LiteRT converts models from PyTorch, TensorFlow and JAX into the .tflite format (Google AI Edge, LiteRT overview).
For generative features, Gemini Nano runs inside Android's AICore system service. The ML Kit GenAI APIs sit on top of it and offer summarisation, proofreading, rewriting and image description, plus a Prompt API for custom prompts. Because the model is shared at system level, your app does not download its own copy (ML Kit GenAI APIs; Android, Gemini Nano). Most of these APIs are labelled beta, and they run only on a published list of recent devices, so treat them as an enhancement rather than a core dependency.
| Need | iOS | Android |
|---|---|---|
| Custom model (vision, audio, classification) | Core ML | LiteRT |
| One model format across both platforms | LiteRT on iOS | LiteRT |
| Built-in on-device language model | Foundation Models (iOS 26+, Apple Intelligence devices) | ML Kit GenAI APIs with Gemini Nano (supported devices) |
| Hardware acceleration | CPU, GPU, Neural Engine via Core ML | GPU and NPU via LiteRT CompiledModel API |
How small does a mobile model need to be?
Small enough to download, load and run without draining the battery or crashing on mid-range phones. Quantisation, which stores weights at lower precision, is the main tool. Google's LiteRT documentation gives the trade-offs for post-training quantisation (Google AI Edge, post-training quantization):
| Technique | Size vs float32 | Speed (per Google) | Runs on |
|---|---|---|---|
| Float16 | About 2x smaller | Enables GPU acceleration | CPU, GPU |
| Dynamic range (8-bit weights) | About 4x smaller | 2x to 3x faster | CPU |
| Full integer (8-bit) | About 4x smaller | 3x or more faster | CPU, Edge TPU, microcontrollers |
On Apple platforms, Core ML Tools supports 8-bit and 4-bit weight quantisation and 8-bit activation quantisation, alongside palettisation and pruning (Core ML Tools, quantization overview).
Every one of these trades accuracy for size. The loss is often small, but it is not zero, and it tends to show up on the hard cases: poor lighting, accents, unusual documents. Keep a test set that reflects real users and compare the quantised model against the original before you ship. Test on the cheapest phone you support, not the one on your desk.
How do you ship and update on-device models?
Decide early whether the model is bundled with the app or downloaded later. Bundling is simplest and guarantees the model is present, but every model change needs an app release and it adds to the install size.
Both platforms support downloading models after install. On iOS, Apple documents downloading a model file and compiling it on the device, which it suggests for reducing app download size, choosing models per user, A/B testing and delivering model updates over the network (Apple, Downloading and compiling a model on the user's device). On Android, Play for On-device AI delivers models as AI packs at install time, as a fast follow or on demand, with variants targeted by device model, system properties or RAM. Individual AI packs can be up to 1.5 GB compressed (Android, Play for On-device AI).
A rollout plan we recommend:
- Version every model and log the version with each prediction, so a bug report names the model as well as the app build.
- Ship new models to a small share of users first and compare quality and crash rates with the previous version.
- Keep the previous model on the device until the new one has loaded and passed a quick self-check.
- Have a fallback path, either a smaller model or a server call, for devices that cannot run the new one.
What goes wrong in on-device ML projects?
The common failures are practical rather than mathematical. The model works on a flagship and stutters on a three-year-old phone. Pre-processing on the device differs from training, so camera images arrive rotated, cropped or in the wrong colour format. Thermal throttling slows a model that ran fine for the first minute. And a built-in language model is missing on half the user base because those phones do not support it.
Each has the same fix: test on real devices, across your actual user mix, with real inputs, before deciding the feature works.
Where should you start?
Pick one feature, decide on-device or cloud using the table above, and prototype it on the lowest-end device you support. That one test answers most of the architecture questions.
For vision-heavy work, our guide to computer vision development goes deeper on building and deploying production models, and our comparison of native and cross-platform app development covers the app side. If you want a team to build it, our mobile app development team builds Core ML and LiteRT features into iOS and Android apps, and our AI services team trains and optimises the models behind them.
