Training a model does require massive data and compute, but evaluating/using an already model (e.g. running a Hot Dog/Not Hot Dog classifier) can be done on mobile hardware. Accelerating this could, for instance, allow this to run in real time on a video feed.

Aside from the runtime difference between training and inference, having on-device ML makes a lot of sense for other reasons.

There can be more guarantees over data privacy, since your data can stay on-device. It also reduces bandwidth as there's no need to upload data for classification to the cloud. And that also may mean it's faster, potentially real time, since you don't have that round trip latency.

This is not necessarily for phones. Lots of (virtually all?) low power IoT devices have ARM cores. There are plenty of environments where the cloud or compute power isn't available.

From what little I understand models are built in the cloud, compressed and evaluated on your phone.

Better hardware probably means less power drain and larger models.

There was some cool stuff about this in Google I/O keynote.

Check out federated learning. https://medium.com/tensorflow/introducing-tensorflow-federat...

Machine learning is like any learning, there's the learning stage and the putting it into practice. ML in the cloud is like researchers coming up with a new way to slice bread, ML on your phone is like the your local baker following the instructions in a new cook book. You still need a skilled baker to follow the instructions.

It doesn't have to be for training the models. You can run versions of trained models locally on your phone. Having a dedicated chip will allow for more snappy calculation of those fancy snapchat filters, language translation, image recognition etc

The way neural networks work is that each perceptron runs a linear equation of the form: w1x1 + w2x2...wnxn. Even if you train the model somewhere else, you still need to hold the weights locally in each perceptron in order to evaluate the model.

This requires hardware in which you can multiply these huge matrices quickly, even if the weights are downloaded from the cloud.

E.g. accurate edge detection algorithms are better done with ML-based mathematics and accompanying optimized transistor layouts, this would enable better augmented reality type graphic additions from things like snapchat selfies to visualizing free-space CAD.