Since modern chips from cell phone application processors to datacenter GPUs all have significantly invested real estate to accelerate matrix operations, it only makes sense to take advantage of it, whether it is optimal for the task at hand.

That is exactly the same path that led us to GPGPU, which in turn derived to using GPUs to accelerate neural nets. In every step of the way it was about repurposing existing hardware that was designed for something else and was thus suboptimal.