News for hackers. Stuff that matters.
Re:Amazon SageMaker – Build, train, and deploy machine learn... by michaelbarton on Wednesday November 29, 2017 @05:58PM (#15811691)
If anyone from AWS is in this forum, could you comment if the custom Docker training in sagemaker can also be used for general optimisation of any dockerised objective function, e.g. bayesian hyper parameter training?

In the blog post example there is this python code:

    def train(
        channel_input_dirs, hyperparameters, output_data_dir,
         model_dir, num_gpus, hosts, current_host):
Would I also write some kind of similar function for scoring the result of the training?

To provide some context, I work in bioinformatics where some of our algorithms have 100s of parameters. This is not ML where we want to classify or predict but rather optimise the parameters for a given objective function. If sagemaker allows general optimisation in an AWS lambda like way, that would be very useful.

Re:Amazon SageMaker – Build, train, and deploy machine learn... by tomfaulhaber on Wednesday November 29, 2017 @06:19PM (#15811827)
Engineer on the SageMaker team here.

There are no restrictions on the types of algorithms that you can optimize using the HyperParameterOptimization service.

SageMaker is designed for machine learning which means it's optimized for algorithms that process a lot of data to develop a model where each run of the algorithm may generate an objective function value (or potentially many such as the value may change during training).

If this structure fits your problem, SageMaker could be useful to you even if your problem isn't strictly "machine learning."

Re:Amazon SageMaker – Build, train, and deploy machine learn... by michaelbarton on Wednesday November 29, 2017 @06:56PM (#15812066)
That's great. Thank you for following up so quickly. Could you point me to where I could read about this, or see an example?

I took a quick search on the documentation and I could see anything on a cursory pass:

http://docs.aws.amazon.com/search/doc-search.html?searchPath...

Re:Amazon SageMaker – Build, train, and deploy machine learn... by tomfaulhaber on Tuesday December 05, 2017 @05:03PM (#15856462)
Sorry, for the very delayed reply here.

The hyperparameter optimization feature is still in preview (though the rest of SageMaker is in general availability). We'll be putting up a page within the next week or so for you to request access.