I upvoted the replies about computational cost and stability, because they raise an important point. But what I was trying to talk about (as you are) was the presentation of the idea.

Linear models are much older than computers, dating back to Gauss at least, and they do not have anything to do with gradient descent.