I've had good results from simplex; i'm a bit sad to hear there isn't some magical algorithm that will give me a massive speedup over it.
I haven't tried automatic differentiation. That seems extremely daunting given the volume of code in my problem. Someone has done some groundwork on AD of the main library i use in my model, but it would still be a serious bit of work.
Optimal knot placement, though seemingly simple, can be a gnarly problem due to degeneracy in many cases. Anecdotally I've found quadratic splines to still be somewhat tractable with exact methods, but polynomial splines with cubic or higher orders can be tricky, especially when they are large-scale.
For derivative-free optimization, you can sometimes get speedups from just adopting a embarassingly parallel algorithm like Particle Swarm. These types of stochastic algorithms aren't necessarily algorithmically better than Nelder-Mead Simplex, but they can explore a larger terrain in parallel, which may help. If you can find some parallel stochastic code, you should be able to spin up a VM on the cloud with lots of cores to help you search the space.
Agreed. CBC/Clp are the best open-source LP/MIP solvers out there. SCIP is better, but it's not fully free.
> For general optimisation, i've used NLopt. It was very easy to use, and has a pretty sensible set of algorithms.
I noticed the omission of IPOPT support in NLopt. IPOPT is one of the best large-scale nonconvex NLP solvers out there (if you have exactly 2nd order derivative information, typically from AD).
> One thing i've learnt is that comparing optimisation algorithms is really hard. One might be better at one problem, one at another. One pretty general axis of variation is problem size: some algorithms scale to orders of magnitude more variables (or constraints etc) than others.
Very true. The academic world has standardized on Dolan-More charts for comparing optimizers on a standard corpus of problems, and those are the closest we have to quantifying general performance. But the no-free lunch theorem applies. There's also a lot of randomness and chance involved, e.g. initialization points can drastically affect nonconvex solver performance on the same problem. So can little things like ordering of constraints or the addition of redundant non-binding constraints (!) for MIPs (this was demonstrated by Fischetti). That's why modern MIP solvers include a random perturbation parameter.
> ML hyperparameter optimisation
Most types of hyperparameter optimization are black-box optimization problem with no analytic derivative information, which much of research in deterministic optimization does not address. Black-box problems typically rely on DFO (Derivative-Free Optimization) methods. DFO methods are even harder to quantify in terms of performance. The Nelder-Mead Simplex method is as good a method as any.