Why on-device inference changes the economics of AI
Where a model runs is an economic decision before it is a technical one.
The question that gets asked too late
Teams building an intelligent feature usually settle the model first and the location second. That is the wrong order. Where inference runs determines what the feature costs per use, how fast it feels, whether it works on a train, and whether the same input reliably produces the same output. Those are product properties, not deployment details, and they are extremely expensive to change once an interface has been designed around them.
We reached the opposite conclusion on our own identification engine and put the whole thing on the device. Not as an optimisation after the fact — as the first architectural decision.
What local execution actually buys
Four things, and only one of them is speed.
- The marginal cost is zero. A hosted model call costs real money every time someone taps the button. For a feature people invoke once a month, that is rounding error. For the central interaction of a product, it is the difference between unit economics that improve with scale and unit economics that degrade with it. Our identification runs against a 500-record catalogue on an 11-dimension vector; it costs the same whether it runs once or ten thousand times.
- It works with no connection. Not degraded — identical. There is no offline mode to design, no queue to drain, no partial state to reconcile when the network comes back.
- It is deterministic. The same description produces the same ranking every time. That sounds obvious until you try to write a test against a hosted model, or explain to a user why the answer changed between two identical queries.
- Latency is a frame, not a round trip. Cosine similarity across a few hundred vectors is arithmetic. There is no spinner to design because there is nothing to wait for.
The last two matter more than they look. Determinism is what makes the behaviour testable, and testability is what let us pin the failure modes below rather than discover them in production.
Where it stops working
This is not an argument that everything belongs on the device. Local execution buys you those four properties in exchange for a hard ceiling on what the computation can be. Matching a description against a structured catalogue is arithmetic over a small vector space. Producing open-ended prose about a result is not, and no amount of architectural preference makes it so.
So we split the product along that line. Identification, recommendation, and profiling — the things users do constantly — run locally. The one capability that genuinely needs a large language model sits behind a server function, gated by entitlement and bounded by spend ceilings. The split is not a compromise; it is the point. Each capability sits where its own economics work.
The guard that matters more than the algorithm. Cosine similarity against an all-zero vector scores zero against every record. That leaves the catalogue in its original order and hands back whichever item sorts first — with a score attached, looking exactly like a real answer. Our engine returns nothing when the input carries no signal, and a test pins that behaviour to null rather than to an arbitrary first row. This is the single most important line in the system, and it is not part of the matching algorithm at all.
How to decide
Ask what the feature costs at a hundred times current usage. If that number is uncomfortable, you do not have a cost problem to optimise later — you have an architecture that punishes you for succeeding, and the time to find that out is before the interface is built around it.
Then ask what the feature does at zero bars of signal. If the honest answer is that it stops, decide now whether that is acceptable, because users will discover it at the worst possible moment and will not describe it charitably.
The next generation of software will be intelligent.
ScentID Solutions is building toward that future — one platform, one integration, and one intelligent solution at a time.
Work With Us