Good point on the fit/predict difference. However, there are some models and techniques (e.g. logistic regression with hashing trick) where the fit and predict steps aren't all that different:

https://adamdrake.com/big-data-small-machine.html

A big benefit for doing everything on-device is that a lot of privacy concerns can be mitigated. I also agree that sending data to a server for learning is an option, and the privacy problems can be addressed with something like client-side feature hashing as I mention in:

https://adamdrake.com/scalable-machine-learning-with-fully-a...

However, doing that in a very power-conscious environment dose pose difficulties with radio usage, which is comparatively power hungry. It's probably a case-by-case situation.

There has been a lot of research into using very low bit depth weights in neural nets, pruning, etc. I am pretty confident that this research, combined with purpose-designed silicon, will allow us to evaluate quite powerful neural net models on embedded systems.

> In a lot of cases, it makes much more sense to send feature data to a remote server that can generate a prediction within milliseconds, and then transmit that prediction back to the device.

It may make sense now, but not having to power up the radio for every decision is a huge gain, as lined out in the article. The current model of dumb (as in ML) devices is coming to an end, see also CoreML from Apple.

The more I learn about machine learning, the more I learn its really all training. After training and a model is available, it seems ready to be commoditized to me. ML as a service seems the only reasonable way the industry can evolve.

And it's coming: https://lobe.ai