News for hackers. Stuff that matters.
Re:Regularized Newton Method with Global $O(1/k^2)$ Convergence by paulpauper on Tuesday December 07, 2021 @12:40PM (#29475647)
I was not aware that this was a problem in need of solving. aren't computers so fast that this is already solved form a practical standpoint. Newton's method (as well as extensions of it to vector analysis) is ancient math. I was not even aware that this is an area of ongoing interest.
Re:Regularized Newton Method with Global $O(1/k^2)$ Convergence by BeetleB on Tuesday December 07, 2021 @01:07PM (#29476015)
> aren't computers so fast that this is already solved form a practical standpoint.

Not in the least. Much of the scientific computation out there is to solve these types of problems. Even if you sped up computers ten-fold you wouldn't be satisfied.

When I was in grad school, the computational teams would run calculations where it would take a few days to get a single data point (they were trying to plot a curve). And they had problems in their head where it would take over a month to get a data point - but of course they weren't pursuing those problems as it was impractical.

Things like computing the derivative - let alone the Hessian - can be quite expensive for some problems.

Re:Regularized Newton Method with Global $O(1/k^2)$ Convergence by owlbite on Tuesday December 07, 2021 @08:21PM (#29480308)
Non-linear optimization is very popular in many fields (non-linear = real world, optimization = find me the least cost/most profit/lowest energy). Nearly all scientific problems can be reformulated as an optimization problem. Training neural networks is a (stochastic) non-linear optimization problem.

Computers are quite fast at doing this in one variable, but typically you're in at least 10s, if not 10,000s of variables, which makes it much much harder.

In dense math, consider n=10,000. Typically a single step involves at least one system solve. Using some matrix factorization approach you're doing O(n*3) operations, assume the constant is ~1 and you have 1 TFlop of computation per step. Modern compute capability of a consumer device is about 1 TFlop/s, give or take. If you're doing (say) 300 steps to reach a solution, you're now at the 5 minute mark for a single moderately sized problem.

Admittedly things like sparse math and iterative solution approaches exist, but so do much bigger problems. And that's before we get to non-convex problems which will often use a convex solver to repeatedly solve subproblems.

Re:Regularized Newton Method with Global $O(1/k^2)$ Convergence by syvon on Tuesday December 07, 2021 @12:57PM (#29475888)
I am unsure if this precise problem needed a better solution, but even if not, this is not how math research works. Finding better solutions to random / useless problems can lead to a better solution to a problem which needs solving. The relevance of the problem is not a consideration.