>Not necessarily, often derivatives are analytically known in ML.

The focus here is largely on deep neural networks. In this domain, the Hessian cannot be computed and SGD (with minor variants) continues to be the golden standard.

Sure, you can build highly reliable supercomputers. But the majority of us are running on google style networks of mostly-reliable boxes. I worked for a company that built distributed ML software, and even at eg hundreds of boxes you will regularly see failures. Customers will be very unhappy unless your code tolerates that.