Join the early list

On the device or in the cloud?

Where a model runs affects speed, privacy, cost, and what happens when the network is down.

Foundation

Running a model on the device or in the cloud On the device, sensor data goes straight to the model and a decision, with no network delay, raw audio and video kept local, and no need to be online. In the cloud, every decision crosses the network. Either way, a common safeguard is to ignore results below a confidence threshold set using real data. On the device Sensor Model runs on the device Decision No network delay · raw audio and video stay local · works offline In the cloud Sensor Network Model runs in the cloud Decision Every decision needs a working connection Safeguard: confidence threshold A classifier always gives an answer, even for inputs unlike its training data below: result ignored above: act on the result threshold Low confidence High confidence Scores are often too high, so set the threshold using real data.
On the device, a model decides without the network; in the cloud, every decision needs a connection. Either way, a common safeguard ignores results below a confidence threshold set from real data.

Key points

  1. Running a model on the device avoids network delay, keeps raw audio and video local, and works offline. Running it in the cloud means every decision needs a working connection.12
  2. A classifier always gives an answer, even for inputs unlike anything it was trained on. A common safeguard is to ignore results below a confidence threshold, but confidence scores are often too high, so set the threshold using real data.34

Where to go next

Sources

  1. ML Kit: on-device machine learning · Google · Project docs
  2. LiteRT for Microcontrollers · Google · Project docs
  3. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (Hendrycks & Gimpel, ICLR 2017) · arXiv / ICLR · Reference
  4. On Calibration of Modern Neural Networks (Guo et al., ICML 2017) · PMLR · Reference