News for hackers. Stuff that matters.
Re:Pattern: A web mining & natural language processing syste... by baltcode on Thursday February 24, 2011 @02:53PM (#2259549)
A recurring meme in terms of frameworks is people keep bringing out tools and some of them disappear, some find niche applications, and some become mainstream. Though I haven't heard of many nlp toolkits (but I'm not in that field).

I want to jump into some basic NLP, but I'd like to stick with one or two toolkits. I had heard of nltk before this, but are there any other comprehensive or sort of succesful frameworks out there one should be aware of? (Either in python or something else)

Re:Pattern: A web mining & natural language processing syste... by gilesc on Friday February 25, 2011 @02:01AM (#2261340)
The best toolkits are probably in Java:

-Stanford's Tagger, Parser, and NLP Core

-Apache OpenNLP

-Lingpipe

Many smaller components are made to be compatible with IBM UIMA (of Watson fame), so they are able to be integrated into a pipeline somewhat easily. For examples of this in biomedical TM, see http://u-compare.org/ .

People will kill me for saying this, but truly: Python's performance isn't adequate for large-scale text mining, _especially_ if you want to do deep/full parsing. Shallow parsing as shown in this package's demo is more feasible.

I personally find NLTK convoluted, but in its favor, it does have readers for a TON of corpora, which is really nice.

Re:Pattern: A web mining & natural language processing syste... by devinj on Friday February 25, 2011 @02:16AM (#2261366)
My friends in the natural language field tell me Python and NLTK are more common than Java. Then again, this is at a sort-of Python-centric university (Toronto).