The hardware requirements for training are much higher than for inference. Roughly:
- Training: Requires massive throughput and compute, so is done on high-end GPUs or TPUs in datacenters (or on researchers' beefy workstations)
- Inference: Compute requirement depends on the size and complexity of the model, but many models can be run on low-end smartphone hardware.
Generally speaking, any time you see "ML hardware" in a consumer device, that hardware is for inference, not training. Apple's Neural Engine is an example, of this. Other current examples include the ARM Ethos-N series and Qualcomm AI Engine, among others. Often this hardware is designed to balance inference performance with power consumption, so that inference can be done efficiently on battery-powered devices.
The hardware requirements for training are much higher than for inference. Roughly:
- Training: Requires massive throughput and compute, so is done on high-end GPUs or TPUs in datacenters (or on researchers' beefy workstations)
- Inference: Compute requirement depends on the size and complexity of the model, but many models can be run on low-end smartphone hardware.
Generally speaking, any time you see "ML hardware" in a consumer device, that hardware is for inference, not training. Apple's Neural Engine is an example, of this. Other current examples include the ARM Ethos-N series and Qualcomm AI Engine, among others. Often this hardware is designed to balance inference performance with power consumption, so that inference can be done efficiently on battery-powered devices.
Training workloads on the M1 use the GPU, not the Neural Engine: https://blog.tensorflow.org/2020/11/accelerating-tensorflow-...