Zephyr314

Do you guys have examples of people actually using this in production? I'm curious how it scales beyond dev.

Very cool to see what used to take a team years to build in a simple, intuitive OSS package. Getting a stack like this up and running in 20 lines of python out of the box would have been unthinkable 10 years ago. Congrats to the team. Can't wait to see where you take this!

Hi, I am one of the founders of SigOpt (acquired by Intel in 2020) and I am happy to answer any questions people may have!

You can also jump right to the code here: https://github.com/sigopt/sigopt-server

Hi, I'm one of the founders of SigOpt (Scott Clark) and still working with the team after our acquisition by Intel in 2020. I am happy to answer questions.

I am incredibly proud that the SigOpt product that has helped thousands of researchers worldwide is now completely open to the broader community.

There are also several papers and blog posts diving into details and tradeoffs of different Bayesian optimization approaches and components here [0]. Example: Covariance Kernels for Avoiding Boundaries [1]

[0]: https://sigopt.com/research/

[1]: https://sigopt.com/blog/covariance-kernels-for-avoiding-boun...

This NVIDIA post goes into extending Bayesian Optimization to multiple metrics [0]. It shows how you can use efficient optimization to find a good Pareto Frontier[1].

[0]: https://devblogs.nvidia.com/sigopt-deep-learning-hyperparame...

[1]: https://en.wikipedia.org/wiki/Pareto_efficiency

hyperopt also uses TPEs [0], this may be a variant/fork of that.

[0]: http://hyperopt.github.io/hyperopt/

This can be a very difficult market, but there are a handful of different projects designed to help with this. In-Q-Tel [0] has been accelerating adoption of new technology for the Intelligence Community for many years and often invests in and helps deploy technology from startups of many sizes. SBIRs [1] can be an effective way to quickly get gov money for research and even culminate in a procurement vehicle for sole source contracts if you get through Phase III. There are also a handful of accelerators like DIU [2] and MD5 [3] designed to help small firms navigate this difficult space. It still isn't easy, but it can help level the playing field a bit when you are just a startup.

[0]: https://www.iqt.org/

[1]: https://www.sbir.gov/

[2]: https://www.diu.mil/

[3]: https://community.md5.net/md5/landing

Which in this case nets you >$1M [0] vs the 80k quoted above. This also assumes no retention grants over 6 rounds.

  [0]: 0.02*0.8^6*$200,000,000 = $1,048,576

I agree that this isn't as common for most end-to-end "AutoML" systems that take a CSV, do light feature engineering/combinations, pipe it into a random forest / GBDT, and then output a model. For many of those approaches there are fewer parameters to tune and you don't get as much lift from tuning them right. Often it is more about quantity of models and ease of use vs quality. I do think that quality will increasingly help though so some tuning will start to be used as the volume, variety, or complexity of the models in these systems increases or the value of the models themselves start to increase.

However, or more complex model pipelines where an expert is probably involved there are lots of tools to help with it and it is quickly becoming automated and less of a "dark art." Some of these tools are built into frameworks like Google/Amazon, some are built into open source platforms (like katib in kubeflow), and others are entire companies building model experimentation platforms (like SigOpt). Many of these can handle everything from traditional hyperparameters like learning rate to architecture parameters to tuning feature embeddings, all at once [1]. I agree with the original author that playing with parameters and doing trial and error optimization of hyper-, architecture, or feature transformation parameters will largely stop happening in the manual way it is done today. All of these methods are orders of magnitude quicker than standard brute-force approaches.

Otherwise, I think you are completely right that there are a ton of aspects of modeling that require domain expertise and nuance beyond pulling a model off the shelf. I think a lot of that comes down to picking the model, picking the data that matters, picking the objective that actually solves the problem for the task at hand, etc. I believe less of that will be high-D non-convex optimization done manually.

[1]: https://aws.amazon.com/blogs/machine-learning/fast-cnn-tunin...

To be fair, the hyperparameter tuning behind these AutoML systems are getting fairly robust. Google bases theirs on Vizier [0]. The Amazon Sagemaker group has people from the gpyopt project [1]. There is also tons of open source projects out there to help for non-enterprise projects [2] [3]. There are also stand-alone companies that help with this explicitly for enterprises [4] (Caveat, I am a founder).

Increasingly I think more time will be spent on the creative/bespoke aspects you mention later in your post, like making sure that you are building a system that actually achieves some business value (vs just getting a better academic-oriented metric result). Hyperparameter tuning is basically trying to do high-dimensional, non-convex optimization on time consuming and expensive to sample functions. Hand tuning is a terrible way to approach this, and is different for each problem as you point out. Experts can leverage their domain expertise and the unique aspects of their data, models, and applications in much better ways.

[0]: https://www.kdd.org/kdd2017/papers/view/google-vizier-a-serv...

[1]: https://github.com/SheffieldML/GPyOpt

[2]: https://github.com/Yelp/MOE

[3]: https://github.com/hyperopt/hyperopt

[4]: https://sigopt.com

While a lot of Bayesian optimization methods use GPs (MOE, Spearmint, BayesOpt, etc) some use TPEs as well (most notably, hyperopt [0]), and some ensemble these methods and others like SigOpt (YC W15) [1] (disclaimer: I'm one of the founders).

I tried to briefly go over the functional interpretation of GPs in this talk [2], although the book by Rasmussen and Williams does a much more thorough job [3] (free online, check out chapter 2 for this approach).

I'm happy to answer any questions about the differences. If you're a student/academic SigOpt is also completely free [4].

[0]: http://hyperopt.github.io/hyperopt/

[1]: https://sigopt.com/research/

[2]: https://www.youtube.com/watch?v=J6UcAdH54RE&list=PLbSwfqjMfj...

[3]: http://www.gaussianprocess.org/gpml/chapters/RW.pdf

[4]: https://sigopt.com/solution/for-academia

For graduate students out there that would rather be doing research than "graduate student gradient descent" (or, high dimensional, non-convex optimization in your head), SigOpt (YC W15) is a SaaS Optimization platform that is completely free for academic research [0]. Hundreds of researchers around the world use it for their projects.

Disclaimer: I co-founded SigOpt and wasted way too much of my PhD on "graduate student gradient descent"

[0]: https://sigopt.com/solution/for-academia

SigOpt (YC W15) has an academic program that allows full access to their optimization platform for academic use [1]. Hundreds of academics around the world have used it and there are already dozens of papers that have been published that have benefited from it [2].

These techniques can be orders of magnitude more efficient than a standard hyperparameter search and really cut down the barrier to entry for these types of results.

[1]: https://sigopt.com/solution/for-academia/

[2]: https://sigopt.com/research/publications/

It would be interesting to see what would happen if you also tried to tune the ensemble towards a specific task in the same way that you could tune a single model.

We've definitely seen that tuning the embedding hyperparameters (along with the others) can have a significant impact on performance. [1]

Additionally, whenever you open up the space of tunable parameters to include the embeddings or feature representations themselves you can usually significantly outperform just a well tuned classifier. [2]

This model seems like it trades off complexity in tuning for complexity of an ensemble, but I wonder what would happen if you tried to have your cake and eat it too and just tuned everything.

[1]: https://aws.amazon.com/blogs/machine-learning/fast-cnn-tunin...

[2]: https://blog.sigopt.com/posts/unsupervised-learning-with-eve...

General employee lockup [1] ends at 6 months usually. A tank right before/at that time could signal what the insiders think and the long term prospects in general.

[1]: https://www.investopedia.com/ask/answer/12/ipo-lockup-period...

Hi, I'm Scott Clark, co-founder of SigOpt (YC W15). We provide hyperparameter optimization as a service.

We have some references to recent articles we've presented at NIPS, ICML, and AISTATS here [1]

We also have a higher level technical blog here [2] (we recently did a series of post on how uncertainty influences optimization in single and multi-metric cases). We've also done some hyperparameter tuning blog posts with our partners AWS [3] and NVIDIA [4].

All of these papers and blogs also contain references to other papers and blogs that can help start you down your literature review. Hopefully this is helpful!

[1]: https://sigopt.com/research

[2]: https://blog.sigopt.com/categories/in-depth

[3]: https://aws.amazon.com/blogs/machine-learning/fast-cnn-tunin...

[4]: https://devblogs.nvidia.com/sigopt-deep-learning-hyperparame...

Hi, I'm one of the founders of SigOpt (YC W15). This is part 3 of a 3 part series we've done on uncertainty in modeling and optimization (Part 1 and 2 here [0] and here [1]). Let me know if you have any questions about this post or SigOpt in general. Javier is a Research Engineering Intern with us and wrote this post with our research team lead, Michael Mccourt. If you're a student looking for internships please check out our careers page [2]. Our platform is also free for academics [3]. You can find more of our research (including NIPS, ICML, AISTATS, etc papers) here [4].

[0]: https://blog.sigopt.com/posts/modeling-with-uncertainty

[1]: https://blog.sigopt.com/posts/bayesian-optimization-with-unc...

[2]: https://sigopt.com/careers

[3]: https://sigopt.com/edu

[4]: https://sigopt.com/research

SigOpt (YC W15) is a related service that performs a superset of these features as a SaaS API (I am one of the co-founders).

We've been solving this problem for customers around the world for the last 4 years and have extended a lot of the original research I started at Yelp with MOE [1]. We employ an ensemble of optimization techniques and provide seamless integration with any pipeline. I'm happy to answer any questions about the product or technology.

We're completely free for academics [2] and publish research like the above at ICML, NIPS, AISTATS regularly [3].

[1]: https://github.com/yelp/MOE

[2]: https://sigopt.com/edu

[3]: https://sigopt.com/research

Hi, I'm one of the co-authors of MOE (it was a fork of my PhD thesis).

We haven't been actively maintaining the open source version of MOE because we built it into a SaaS service 4 years ago as SigOpt (YC W15). Since then we've also had some of the authors of BayesOpt and HyperOpt work with us.

Let me know if you'd like to give it a shot. It is also completely free for academics [1]. If you'd like to see some of the extensions we've made to our original approaches as we've built out our ensemble of optimization algorithms check out our research page [2].

[1]: https://sigopt.com/edu

[2]: https://sigopt.com/research

Thanks! I'm one of the co-founders of SigOpt (YC W15).

You hit the nail on the head. We've been trying to promote more sophisticated optimization approaches since the company formed 4 years ago and are happy to see firms like Google, Amazon, IBM, and SAS enter the space. We definitely feel like the tide of education lifts all boats. Literally everyone doing advanced modeling (ML, AI, simulation, etc) has this problem and we're happy to be the enterprise solution to firms around the world like you mentioned. We provide differentiated support and products from some of these methods via our hosted ensemble of techniques behind a standardized, robust API.

We're active contributors to the field as well via our peer reviewed research [1], sponsorships of academic conferences like NIPS, ICML, AISTATS, and free academic programs [2]. We're super happy to see more people interested in the field and are excited to see where it goes!

[1]: https://sigopt.com/research

[2]: https://sigopt.com/edu

Hi, I'm one of the founders of SigOpt (YC W15). This is part 2 of a 3 part series we'll be doing on uncertainty in modeling and optimization (Part 1 here [0]).

Let me know if you have any questions about this post or SigOpt in general. Gustavo is a Research Engineering Intern with us and wrote this post with our research team lead, Michael Mccourt. If you're a student looking for internships please check out our careers page [1]. Our platform is also free for academics [2]. You can find more of our research (including NIPS, ICML, AISTATS, etc papers) here [3].

[0]: https://blog.sigopt.com/posts/modeling-with-uncertainty

[1]: https://sigopt.com/careers

[2]: https://sigopt.com/edu

[3]: https://sigopt.com/research

Hi, I'm one of the founders of SigOpt (YC W15). This is part 1 of a 3 part series we'll be doing on uncertainty in modeling and optimization. Let me know if you have any questions about this post or SigOpt in general. Mislav was recently a Research Engineering Intern with us and wrote this post with our research team lead, Michael Mccourt. If you're a student looking for internships please check out our careers page [1]. Our platform is also free for academics [2]. You can find more of our research (including NIPS, ICML, AISTATS, etc papers) here [3].

[1]: https://sigopt.com/careers

[2]: https://sigopt.com/edu

[3]: https://sigopt.com/research