simonblanke

I am glad you like it. Hyperband relies on early stopping of the model. So it is something you would do inside the objective function by yourself. I have plans to add some helper functions for early stopping in the future.

Thank you very much :-) yes you can put anything you want into the search space. Even pandas dataframes, numpy arrays or classes. Here is an example: https://github.com/SimonBlanke/Hyperactive/blob/master/examp...

Check out the Neural Architecture Search Tutorial here: https://nbviewer.jupyter.org/github/SimonBlanke/hyperactive-...

Neural Architecture Search is just one of many optimization applications you can work on with Hyperactive. Check out the examples in the official github repository: https://github.com/SimonBlanke/Hyperactive/tree/master/examp...

Yes it is quite easy to switch algorithms via the "gpr" parameter. You just have to write a wrapper class. I am currently working on a repository that discusses how to do that in detail: https://github.com/SimonBlanke/surrogate-models I will upload some examples (GPy and Gpflow) within the next 24 hours.

I think those wrapper-classes will already answer some of your questions. If you have additional or unanswered questions you can open an issue in the Grad-Free-Opt repository.

Yes Hill Climbing + some of its variants are featured in this package.

Unfortunately NLopt cannot be included into Gradient-Free-Optimizers. But i would like to implement multiple algorithms that are also preset in NLopt.

I think it will help my understanding of optimization algorithms if i implement them myself. I currently have my eyes on the Downhill-simplex, Direct and Powell's Method. But they are quite different from what i have already implemented, so this could take some time.

Your parallel computing approach sounds intriguing! Could you provide an example script? I would like to look into this. If you like you could open an issue as a feature request and provide a code snipped there.

I thought about a table in the readme that shows some kind of metric for each optimizer that describes its performance. I will look into that.

I will look into this algorithm. Thanks for the suggestion. I have some basic explanations of the optimization techniques and their parameters in a separate repository: https://github.com/SimonBlanke/optimization-tutorial

But there is still a lot of work to be done.

Two very interesting questions! I should work soon on a comparison of Hyperactive and GFO to other packages. If some important features are missing, maybe i could add them.

I will also look into Dlib. If you like you can open an official issue in my repository. We could discuss why it is useful and if it should be implemented.

Gradient-Free-Optimizers has a type of genetic algorithm: Evolution Strategy

Evolution strategy is included. The "Covariance matrix adaptation" is for making this algorithm work for continuous search spaces. But gradient-free-optimizers has discrete search space.

Yeah this is a warning from sklearns gaussian process regressor. Sklearn probably wants you to increase the "n_restarts_optimizer"-parameter of the gpr, but from my experience this warning does not correlate with bad results from Gradient-Free-Optimizers.

Sklearn warnings are often difficult to silence. Just google "sklearn suppress warnings" to get a code snippet for this problem.

You could be right. I must confess, that i have a (probably) very narrow understanding of typical optimization problems. Most of the objective functions i optimize have machine learning algorithms in it (to optimizer hyperparameters). Depending on the evaluation those can have low noise.

If you like you could provide other use cases and applications for optimization algorithms.

I am currently working on the Nelder-Mead algorithm. I did not realize that it is that popular. This gives me motivation to implement it soon ;-)

If there are more "must have"-algorithms you could open an issue in Gradient-Free-Optimizers.

Okay that is interesting. You could realize that by making a restricted area in the search space by returning np.nan in the objective function for those cases. Gradient-Free-Optimizers can handle np.nan and np.inf just fine.

Maybe you could do something like:

If a+b>=1: return np.nan else: return score

I had the same excitement for tpot! That project was the reason i started creating mine.

Gradient-Free-Optimizers is a lightweight optimization package that serves as a backend for Hyperactive: https://github.com/SimonBlanke/Hyperactive

Hyperactive can do parallel computing with multiprocessing or joblib, or a custom wrapper-function. Parallel computing can be done with all its optimization algorithms. You could even pass a gaussian process regressor like GpFlow to the Bayesian Optimizer (GFO and Hyperactive) to get GPU acceleration.

Also: Gradient-Free-Optimizers is basically just the optimization-backend for a much larger project of mine: https://github.com/SimonBlanke/Hyperactive

I think Random Restart Hill Climbing is good if your objective function evaluates fast (simple functions). It does not get stuck in local optima and also does local search very well.

Bayesian Optimization is very good if your objective function takes some time (> 1 second) to evaluate. If you look at the gifs in the readme of Gradient-Free-Optimizers you will see how fast it finds promising solutions. It is almost scary how good it is (even compared to other smbo).

I am currently working on the Nelder-Mead optimiser. I will also look into "BOBYQA". I am always searching for interesting algorithms that my users need. If you have more suggestions you can open an issue in the repository.

I am glad you like my project. The search space is not continuous. But you can make the steps as small as you want. Like: np.arange(0, 10, 0.00001)

I am not sure i understand your second question. The entire search space is always a N-dimensional cube. So you just define a start and end point and the step size in each dimension.

I just saw, that my project got a lot of new stars on github. A surprise to be sure, but a welcome one.