I checked the guy’s LinkedIn, he has 5 years of econometrics and data science education. I am honestly perplexed how anything in this article could come off as new to someone with such background.
[2] https://docs.scipy.org/doc/scipy/reference/tutorial/optimize...
But what works for traditional statistics (like matrix inversion) tends to have the wrong big-O for big data. Would be nice though if there were a website somewhere that summarized the state of the ML-applicable optimization field with links to papers in order of increasing complexity. Instead there are like a billion websites rehashing basic SGD that seem to dominate search results.
Although I guess most of the easy to use methods like Adam are already baked into all the frameworks.
We seem to be returning to a world where the only reliable way to find information is to stumble upon someone's webpage (e.g. [1]) who has curated a list of links.
> The objective function for linear regression is quadratic (and convex). This explains why Newton's method finds the optimum in one step--the local quadratic approximation turns out to be a global approximation!
> Notice, however, that Newton's method requires us to invert the Hessian matrix (f''), which is expensive in general. I don't think autograd software can get around this limitation.
> The KeepStepping optimizer is performing "gradient descent with line search", which is a more Googleable term.
For more about optimization for solving linear systems (as is the case for linear regression), I recommend Shewchuk 1994, "Conjugate Gradient without the Agonizing Pain" [1], which has some nice geometric insight.
[1]: http://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradi...