"Should the AI run on the device or in the cloud?" is the question I get most from developers adding a model to a mobile app, and the honest answer is that I have shipped it three different ways in three of my own apps, and each was right for that app.
- ORA sends every request to a backend that routes it to one of twenty-plus hosted models. Nothing runs on the phone except the UI.
- Ome keeps all user data on the phone, encrypted, builds a small context locally, and sends that single block to a hosted model for one stateless request.
- AeroPark runs YOLOv8 and OCR on a single machine with a GPU, with no network in the inference path at all.
The architectures differ because the answers to five questions differ. The questions are the useful part.
1. What is the model's job, and how big is a model that does it well?
This one is decided by the state of the art, not by you. Generating video, cloning a voice, or answering open questions with a frontier model means the cloud, because no phone runs those models at a quality anyone would pay for. ORA's whole promise is "the best model for each task", which makes it a cloud app by definition.
Detecting objects in a frame is a different story. Small detection models have run on phones for years, and AeroPark's YOLOv8 runs comfortably on a laptop GPU. If the job is classification, detection, OCR, speech-to-text for one language, or embedding a sentence, start by assuming it can run locally and make the cloud prove it is necessary.
The question to write down: what is the smallest model that does this job at the quality the product needs? If that model fits on the device, the rest of this list is about whether you want it there.
2. Where does the data have to be?
This is the question that decided Ome. The product promise is that your tasks and notes never sit on a server. The assistant, though, needs a capable language model, and the capable ones live in the cloud.
The resolution was not "on-device" or "cloud". It was on-device data, cloud model, minimal bridge: build a short, readable context on the phone, send only that for one request, store nothing on the server. The data stays local; the intelligence is borrowed. I wrote up the details in the Ome post.
Write down which data is allowed to leave the device, and in what form. Raw data, a summary, or nothing. The answer often points at a hybrid rather than either pure option.
3. What latency does the product need, and what network can it count on?
AeroPark reads plates from a drone at 30 frames per second. A round trip to a server per frame would have made that impossible regardless of the model. Any product that reacts to a camera in real time belongs on the device or on a machine next to it.
ORA's users wait a few seconds for an answer and expect to. A chat response at two seconds feels fine; a camera overlay at two seconds feels broken. Match the architecture to the feeling.
Then the network. A productivity app opened on a plane should still work. Ome's assistant does not, but everything else in Ome does, because the assistant is a feature on top of a local-first app rather than the thing holding the data. Decide which features are allowed to disappear when the network does, and make sure the core is not one of them.
4. Who pays per request?
Cloud inference has a bill per call. On-device inference has a bill per download, paid in app size and battery.
With ORA the per-request cost is the business model, so the router tracks spend per user and picks a cheaper model when the budget says so. That is a feature of the cloud: you can change the cost of a request tonight without a release.
On the device the cost is fixed and mostly invisible until a review says the app drains the battery. Small detection models are fine. Anything that keeps the GPU busy continuously, as AeroPark does, is not something I would ship inside a phone app without a very good reason and a very visible toggle.
Write down the expected requests per user per day and multiply. If the number scares you, that is useful to know before you build the cloud version.
5. How often will the model change?
Cloud models change under you whether you like it or not, and you can change them whenever you like. That is the strongest argument for the routing layer in ORA: a new model is a config change, not an app release.
An on-device model changes when the app updates, which means store review and a long tail of users on old versions. For a detector trained once on a stable task that is fine. For anything that rides the frontier it is a liability.
The quiet middle option is on-device data with a cloud model, which gets the data guarantee of local and the update cadence of cloud. It is the reason Ome's assistant has improved several times without a release.
The short version
| Question | Pushes toward device | Pushes toward cloud |
|---|---|---|
| Smallest good model | Fits in tens of MB | Needs a frontier model |
| Data | Must stay local | Already server-side |
| Latency | Per-frame, camera, offline | Seconds are fine |
| Cost | Fixed, battery | Per request, adjustable |
| Model change | Rare, stable task | Frequent, improving |
If three of the five point the same way, go that way. If the data column says device and the model column says cloud, build the bridge: local data, a small readable context, one stateless request, nothing stored. It is the configuration I keep coming back to, and the one I would start with for most new apps that want an assistant without a privacy policy nobody reads.
You can try all three on a phone: ORA for the cloud version and Ome for the hybrid. AeroPark is a private experiment, written up here.