'...it should be trivial...'

...but incredibly time consuming. Developing bug free, high performance op implementations takes time and elbow grease. It's worth comparing the gradual progress of tf lite (inference on arm devices) and the tpu architectures: rolling out a full set of ops for a new platform is a multi year project. When it's only partially done, you get inconsistent support and weird performance bottlenecks; so maybe a minimal mnist model is fast, but many other things pass back to cpu for lots of intermediate ops, killing performance.