In Python there is a wonderful library called the Natural Language Toolkit (NLTK) available free and open source at http://www.nltk.org/.

With NLTK you can build classifiers, decision trees, and train/predict with bayesian classifiers similarly to Google's Prediction API examples. It's pretty easy to get started, and it's code that you run locally, so there is no network traffic.

I use it on http://www.protopub.com for classifying rss feed stories based on user feedback, so Protopub can recommend future stories that you might like. NLTK is far easier than rolling your own classifiers, but even that is not too difficult. See the O'Reilly book Programming Collective Intelligence.

There's also Weka, which can use almost exactly the same file format that Google is using, and do the same kind of things (though perhaps with different algorithms). It's pretty pleasant.

http://www.cs.waikato.ac.nz/ml/weka/

I recently used Weka to create a simple rules-based fraud model for Kongregate. It worked very well, and had a lot of options for algorithms. The UI is a little weird, but it's worth checking out.

Can you please try our news recommendation service and let me know how it stacks up against your site in terms of (perceived) accuracy? The site is http://www.euraeka.com I can provide more info if you need it. Email me haidut (at) gmail (dot) com. I just need some feedback from someone who has built something similar/related.

Python's Reverend Thomas library is a naive Bayes classifier library. You can do the language guessing and other basic classifications with it: http://divmod.org/trac/wiki/DivmodReverend

You scratched the itch I procrastinated to scratch. Others (à la Postrank) tried, you just seem you have tried better! 1 Kudo :) EDIT: Ah, you're the guy behind Raphaël Vector lib, have one more Kudo!

Raphaël was written by Dmitry Baranovskiy. I only made very minor contributions to the library (animation easing, which are pretty ubiquitous in the examples now) and presented the library at the SVG Open 2009 at Google.

I made Protopub to scratch the itch I think a LOT of us have. I am about a month away from a v1.0, and that's when I'll announce it on HN. Until then, I'm tweaking AI algorithms, fixing UI bugs, and making sure the back-end can handle the more than moderate traffic that HN will send. The few users I get from posts like this are enough to do some basic testing.

Super stuff. I've invested in a Dutch start-up that had this vision about 3 years ago, but got stranded along the way and pivoted in to being a website building company because they found they could make more money in the short term that way (mistake...).

Anyway, I have some appreciation for the difficulties you must have encountered, and it doesn't please me but it will please you to know that at least from them you won't be having much competition.

Is it ok to start using your service? (not from an industrial espionage point of view but because it is useful!)

Absolutely you can start using it. I evolve it every single day. I use Protopub exclusively as my feed reader, and have been for about a month.

Right now, Protopub is an experiment, but it also serves as a beacon to other likeminded hackers in NYC, where I live, that I am interested in meeting others who want to create unique and technically savvy projects. It has done a good job of doing exactly that so far.

Weirdest thing, when I read HN through protopub it wants me to log in again ?

edit: hm, protopub.com proxies all requests ?

I wish you the best! You'll perhaps have some competition if it still itches in spite of your job, after Broodwar AI Competition is over though. :)

there is nothing out there which would scale to terabytes out-of-the-box

VFDT (Very Fast Decission Trees), maybe?

You can also run those distributively without much problems.

Original paper: http://www.cs.washington.edu/homes/pedrod/papers/kdd00.pdf

You should also look on http://www.cs.washington.edu/dm/vfml/