News for hackers. Stuff that matters.
Re:Show HN: A radically more scalable alternative to Bayesia... by chroem- on Friday January 26, 2018 @07:01PM (#16243934)
Hi there,

I'm currently away from my computer but I can produce a comparison animation in a few hours. While I am biased, I strongly suspect that Simple will have superior sample efficiency and at least similar performance.

Re:Show HN: A radically more scalable alternative to Bayesia... by captaindiego on Saturday January 27, 2018 @10:57AM (#16246635)
Cool, I'd be really interested in seeing that and might try out myself later. It'd be really interesting to see a comparison of the two on optimization problems with a large number of dimensions which stock CMA-ES takes a while (and a lot of samples) to get "warmed up".
Re:Show HN: A radically more scalable alternative to Bayesia... by chroem- on Saturday January 27, 2018 @08:48PM (#16249247)
I ran into some issues when comparing the two algorithms, but here is a comparison with CMA-ES.

https://github.com/chrisstroemel/Simple/blob/master/cma-comp...

Simple is a constrained optimizer while CMA-ES is unconstrained, so I ended up choosing a sigma value such that the three sigma cutoff could be considered its faux "optimization domain." I gave the two algorithms equal areas to operate on, however CMA-ES is very sensitive to where it's initialized at. If I had started it near the global optimum it would have instantly converged without performing any optimization at all, so then there was also the issue of deciding on a "fair" starting location. Trying to get around this, I briefly tried the comparison on the Griewank function from -5:5, however CMA-ES showed nonconvergent behavior.

CMA-ES seemed to have a higher startup cost than Simple, but very similar per-iteration runtime performance. Sample efficiency ranged from totally nonconvergent to instant convergence without global search, depending on where it was started.

If you have any suggestions for a better testing methodology I would really appreciate them, because I'm still not quite satisfied with this comparison.

Re:Show HN: A radically more scalable alternative to Bayesia... by captaindiego on Monday January 29, 2018 @09:33AM (#16257682)
Thanks that's quite interesting. Your results of seeing higher startup costs definitely mirror my experience.

When I need constraints on CMA-ES for real world stuff I've been using a simple equation to map the output of the parameters before using them:

  y = a + (b - a)*(1 - cos(pi * x / 10)) / 2
Where a is min, b is max and x is the unconstrained input value. (I saw some random paper recommending this years ago and have been using it ever since).

If I get a chance today I'll try and run your code and play around a bit. Really cool stuff!