This article explores what on-device artificial intelligence is, how it works, and how it is transforming mobile capabilities on iOS and Android.
Understanding on-device AI
On-device AI runs artificial intelligence algorithms locally without relying on cloud servers. While this article focuses on mobile, the same principles apply to wearables, IoT devices, and other edge systems.
Many smartphones and wearables now include specialised AI chips that accelerate and improve the efficiency of this processing. It also helps protect user privacy and lets apps respond instantly, even offline.
Examples of on-device AI in action
- Camera apps enhance photos on the fly, on-device assistants talk to a user and recognise speech.
- Watches perform fall detection and activity recognition.
- Smart home cameras integrate offline person and motion detection.
- Industrial embedded systems detect anomalies in the production process.
Benefits of on-device AI
Sensitive data (photos, health information, or financial details) stays on the device. This protects end-user privacy and helps companies comply with GDPR and HIPAA.
Processing data locally decreases dependence on expensive cloud servers and bandwidth. Companies can now offer AI features to millions of users without massive cloud backend costs.
Without sharing personal data externally, devices adapt to user habits and preferences, offering tailored recommendations.
On-device AI doesn’t need a constant internet connection. Apps continue to work in locations with limited or no connectivity.
Evolution of on-device AI in mobile
2011–2016: Feature-driven mobile ML
On-device AI dates back to 2011, with Siri on the iPhone 4S (A5 CPU). While voice recognition was handled in the cloud, signal pre-processing was performed locally on the device.
Around the same time, Google began developing DSP-based machine learning pipelines on Android using Qualcomm’s Hexagon architecture.
By 2014, iOS 8 had introduced several on-device AI features, including predictive keyboard, face detection and activity classification via motion coprocessors.
Meanwhile, Google applied ML-based clustering in Google Photos. At this stage, devices could run small ML models on mobile GPUs. However, memory and library limitations restricted the size and complexity of workloads.
2017–2022: Platform acceleration and dedicated NPUs
2017 marked the start of modern on-device AI:
- Apple introduced the A11 Bionic with a Neural Engine powering Face ID, Animoji, and other features. The Core ML and Vision frameworks made it easier for developers to run AI models on-device.
- Google launched TensorFlow Lite, NNAPI 1.0, and Pixel Visual Core to accelerate AI for computational photography.
- Huawei released the Kirin 970 with a dedicated NPU. It enabled camera scene recognition, image processing, battery optimisation, and on-device translation.
2018–2019 saw expanded AI frameworks and capabilities:
- Apple Core ML 2–3 added support for on-device model training.
- Google ML Kit introduced text recognition, image labelling, barcode scanning, entity extraction, and pose detection.
- The Pixel 3 and 4 improved computational photography and added full-on-device speech recognition.
- Updates to NNAPI 1.3 and TensorFlow Lite enhanced quantisation, RNN support, GPU acceleration, and custom model deployment.
2020–2023: generative and contextual on-device AI
Apple improved on-device capabilities with:
- Enhanced dictation and predictive typing
- Natural language processing
- Live Text (OCR using Vision Transformers)
- Personal voice creation
- Sensitive content detection
The A17 Pro Neural Engine reached over 20 trillion operations per second, enabling faster, more complex AI tasks to run directly on the device.
Google introduced ML Kit, offering entity extraction and Smart Reply, digital ink recognition, face mesh and keypoint detection, selfie segmentation.
Google also laid the foundation for on-device generative AI with Gemini Nano and LiteRT. Pixel 6 and 7 devices included Tensor SoCs optimised for on-device summarisation and improved voice typing.
2024: breakthrough year for on-device LLMs
- Apple introduced Apple AI (3B+ parameter large language models) for rewriting, summarisation, proofreading, image generation, and semantic search, running locally on A17 Pro-class SoCs.
- Google launched Gemini Nano for summarisation, text rewriting, smart replies, and on-device GenAI tasks.
- ML Kit expanded with a GenAI API that supports chat-style inference, summarisation, and image question-answering.
- LiteRT supported efficient on-device training and fine-tuning of AI models.
2025-2026: platform unification and LLM standardisation
- Apple gradually improves LLM support, integrating on-device personal context graphs with Private Cloud Compute.
- Android 15 standardises Gemini Nano for different types of devices. It allows compact LLM bundles with LiteRT for many GenAI features.
- On-device LLM inference APIs, generative text and vision tasks, and local adaptation via training became widely available.
- Device vendors introduced specialised NPUs optimized for 3–8B-parameter models, with advanced quantisation and memory-efficient runtimes.
Platform considerations: iOS vs Android
| iOS | Android |
|---|---|
| Strength: Stability and privacy | Strength: Flexibility and scale |
| Consistent hardware with mature frameworks (Core ML, Vision) and robust security | Diverse hardware support with ML Kit and LiteRT for customized AI features |
| Best for: Premium consumer apps | Best for: Mass-market applications |
What’s next: On‑device AI as core mobile infrastructure
We already see a shift from running isolated, task-specific models to acting as a continuous, context-aware intelligence layer. Advances in specialised hardware, model efficiency, and software tooling will make local inference the default rather than the exception for many AI use cases.
It will not take long for the budget phones to get NPUs and catch up with the flagship models. Techniques such as quantisation, distillation, pruning, and modular model design will allow devices to run multiple models simultaneously.
Personalisation will increasingly happen on the device, with models fine-tuned locally to individual users.
The hybrid edge–cloud architectures will mature. On-device AI will handle real-time perception, filtering, and decision-making, while the cloud will be used selectively for heavy training, long-term analytics, and cross-device coordination.
Finally, on-device AI will become more autonomous and proactive. With the rise of an agentic AI approach and blending of context and sensor fusion, the user interaction will become more humane and unlock new experiences.
Related Insights
The breadth of knowledge and understanding that ELEKS has within its walls allows us to leverage that expertise to make superior deliverables for our customers. When you work with ELEKS, you are working with the top 1% of the aptitude and engineering excellence of the whole country.
Right from the start, we really liked ELEKS’ commitment and engagement. They came to us with their best people to try to understand our context, our business idea, and developed the first prototype with us. They were very professional and very customer oriented. I think, without ELEKS it probably would not have been possible to have such a successful product in such a short period of time.
ELEKS has been involved in the development of a number of our consumer-facing websites and mobile applications that allow our customers to easily track their shipments, get the information they need as well as stay in touch with us. We’ve appreciated the level of ELEKS’ expertise, responsiveness and attention to details.
