News for hackers. Stuff that matters.
Re:How to Escape Saddle Points Efficiently by cool_username on Friday September 01, 2017 @11:48AM (#15149489)
> Our sharp rate depends on a key observation — although we don’t know the shape of the stuck region, we know it is very thin.

Oh... really? :)

(After 12 years I finally get an excuse to show a fun side project I coauthored during my PhD...)

http://graemebell.net/pubs/taros05-bl-embedded-preprint.pdf

Check out Figure 5 / Section 3.4

The rest of the paper is an introduction to why saddle points can be surprisingly problematic for people using potential fields (neural nets, game/robot navigation). Hope someone finds it interesting.

Re:How to Escape Saddle Points Efficiently by Eridrus on Friday September 01, 2017 @12:27PM (#15149926)
These results seem important for nonconvex optimization in general, but for ML applications where we usually use stochastic/batch gradient descent, I wonder if the the stochasticity adds enough perturbation for this to not really be that useful.
Re:How to Escape Saddle Points Efficiently by Cacti on Friday September 01, 2017 @12:45PM (#15150137)
Well, in general, no, there is no perturbation method large (or good) enough to get out of saddle points, including via stochastic gradient descent. It might work in a particularly specific problem, but not in general.
Re:How to Escape Saddle Points Efficiently by Eridrus on Friday September 01, 2017 @12:50PM (#15150179)
Fine, but I mostly care about ML applications, where I'm wondering if this is expected to help at all.
Re:How to Escape Saddle Points Efficiently by Cacti on Friday September 01, 2017 @01:05PM (#15150326)
ML is far too broad a category. Like I said, it will depend on the problem/function.
Re:How to Escape Saddle Points Efficiently by robrenaud on Friday September 01, 2017 @03:10PM (#15151559)
How does what you say agree/disagree with the authors paper?

The authors bound derivatives and 2nd derivatives, which excludes particularly nasty functions?