News for hackers. Stuff that matters.

A comment on Accelerated PyTorch Training on M1 Mac. Parent comment

Re:Accelerated PyTorch Training on M1 Mac by Salgat on Wednesday May 18, 2022 @08:44PM (#31429354)
For higher feature counts the real concern is saddle points rather than minima, where the gradient is so small that you barely move at all each iteration and get "stuck".
[ Parent | on HN ]
Re:Accelerated PyTorch Training on M1 Mac by timomo on Thursday May 19, 2022 @02:25AM (#31431354)
To add here: for a local minimum to occur all those dimensions (or features) need to increase. This is highly unlikely for modern NNs where you have millions of dimensions. If one of the dimensions is going down but the rest up, you have a saddle point. Since you go down only one (or few) dimensions it takes longer.
[ Parent | on HN ]