News for hackers. Stuff that matters.
Re:A Programmer's Guide to Data Mining by SeppoErviala on Sunday December 08, 2013 @07:52AM (#6869598)
Check out gensim if you want to do topic modeling or similarity comparisons in Python.

http://radimrehurek.com/gensim/

It has good implementations of various algorithms, some of which support streaming or dirstribution, and it allows loading and dumping data in various formats.

I've used it for building content based recommender using tf-idf, lsi and similarity index. After the index is built, queries to it are really fast. It can handle quite large corpuses with little memory.

Re:A Programmer's Guide to Data Mining by sbrother on Sunday December 08, 2013 @12:31PM (#6870342)
Second this, I'm surprised you don't read more about it here. We use it in production to recommend image searchterms based on unstructured text, and it performs better with a few lines of python code than anything our team could write in a lower level language in months. It's REALLY fast once you've built an index.

The reason for that is a pretty epic list of dependencies (have fun explaining why the prod boxes need a fortran compiler), but in terms of efficiency and speed of development it's an obvious choice.

Re:A Programmer's Guide to Data Mining by Radim on Sunday December 08, 2013 @08:45PM (#6872465)
:-)

Hopefully the SciPy & BLAS dependencies will only get easier to install from now on... Continuum Analytics received shit loads of money and some of it is going towards better scientific Python packaging, I believe.

Re:A Programmer's Guide to Data Mining by hnriot on Sunday December 08, 2013 @11:39PM (#6872840)
gensim is awesome, it abstracts very complex algorithms into extremely simple function calls. The models.HdpModel class is very powerful.