News for hackers. Stuff that matters.
Re:Algorithm Development is Broken by chubot on Wednesday February 05, 2014 @03:40PM (#7186643)
Hm, sounds interesting but vague. I don't really understand how it will work.

How is data for the algorithms provided? A lot of times it is big. And messy, and proprietary. There seems to be an implicit assumption that you can just plug different algorithms to different data sets. But I can't think of a project where that has been the case.

I also doubt that "algorithms" are the bottleneck in a lot of projects. I'm not an expert, but I have some personal experience to back up what people say about "data trumping algorithms" (e.g. Peter Norvig and others have written about this)

I would like to hear about some more concrete examples / success stories. "Algorithms" is just too vague. I think if this becomes successful, it will be by first narrowing it to a particular domain, and then generalizing it again.

Re:Algorithm Development is Broken by platypii on Wednesday February 05, 2014 @04:03PM (#7186777)
Sorry for being vague, that wasn't our intention, so much as to focus on laying out the problem that we've experienced personally as algorithm developers. We will be posting more blog posts with much more detail about the system in the near future.

The data question is huge for us. We have built our system on top of Hadoop + Spark in order to handle large amounts of data, and be able to apply computations to it. You're absolutely right that data is very heavy, and often the limiting factor, so we are doing everything we can to both get data into the system (currently with a focus on streaming data, since that gets around the heavy data problem), and getting algorithms to where the data is.

As far as specific algorithms, there are many that could apply. My personal experience is in machine learning algorithms, so I'm personally biased toward those. There are numerous algorithms like Classification, Clustering, Optimization, Anomaly Detection, etc. which are very CPU-intensive. We will be doing a blog post soon with a demo of live algorithms already in our system to give more concrete examples.

Re:Algorithm Development is Broken by heurist on Wednesday February 05, 2014 @06:48PM (#7187680)
Could you return the actual code of the algorithm that could be loaded and used during runtime (for some languages if the user wants it)? Seems like having the user send in the data and running an algorithm as a service would take a lot more time and effort for everyone involved.