News for hackers. Stuff that matters.
Re:Software design patterns for Machine Learning R&D by textminer on Wednesday August 15, 2012 @03:43AM (#4385031)
Hardest part of implementing a high-functioning production machine learning stack for me isn't the idea-articulating, prototyping, iterating, then polished refactoring. It's knowing when to go from a quick-producing language like Python/MATLAB/Julia to something painfully-written but smooth like C++ or into a scalable Elastic MapReduce or Mahout process (the former of which, sure, is language-agnostic).

You can only spend so much time optimizing on memory/CPU-times with smart data chunkings or low-dimensional representations or approximation operations. EC2 time and space is relatively cheap, but Python on a single machine with the multiprocessing module can only speed up by a multiple of < [# of Cores]...

Re:Software design patterns for Machine Learning R&D by beambot on Wednesday August 15, 2012 @04:31AM (#4385125)
You should checkout PiCloud [1] or MrJob [2]. Both of these seek to make MapReduce dead simple using Python. More importantly, you can do all your testing on a Desktop PC. Then when you need real horsepower, you just tell it to "spin up on EC2."

Disclaimer: I have yet to use either, but I've heard good things.

[1] http://www.picloud.com/ [2] http://musicmachinery.com/2011/09/04/how-to-process-a-millio...