News for hackers. Stuff that matters.
Re:Natural Language Processing with Python by gilesc on Thursday September 29, 2011 @01:44PM (#3053449)
NLTK is great for _learning_ NLP, but Python is much too slow for scalable deep NLP (by which I mean tagging and parsing, as opposed to TF-IDF etc). Also parallelization can become a problem because of the GIL. It's a real shame they chose Python actually, because otherwise it's a superbly structured, documented, and maintained project.
Re:Natural Language Processing with Python by Radim on Thursday September 29, 2011 @02:26PM (#3053632)
Hmm, I think Python was an excellent choice; what other platform would you suggest? IMO being "superbly structured, documented and maintained" is not a magical property acquired by luck, but rather connected to the platform of choice.

Btw for performance, whenever pure Python is indeed "much too slow" (profile?), there's the option of C extension modules. The NumPy or SciPy libraries are good examples: used in hardcore numerical computing aka the epitome of I-NEED-IT-TO-RUN-FAST!, but still Python.

And not to nitpick ;) but GIL only affects multi-threading; other modes of "parallelization" are reasonably straightforward and some even built-in (import multiprocessing).

Re:Natural Language Processing with Python by cf on Thursday September 29, 2011 @03:46PM (#3053945)
Yes, that is how you make fast libraries in Python. But, nltk isn't written using C extension modules. All of this NLP is done in pure python. You could rewrite what needs to be fast with C extensions, but then what's the point of using nltk in the first place?

Nltk was never intended to be a way to do production-grade natural language processing. It's primary objective has been to teach users natural language processing with clear, well-commented code and documentation. If this isn't your situation, please use something else.

Re:Natural Language Processing with Python by andreasvc on Friday September 30, 2011 @06:07AM (#3056378)
What's the point? That half of your code base has already been written for you. Rewriting performance critical parts is a lot of work, and not having to rewrite a corpus reader, tree transformations, or evaluation procedure is an advantage; aside from being an excellent prototype platform. With Cython you can seamlessly combine Python code such as that from NLTK with your own optimized code. This was indeed never the intention of NLTK, but I have found the general idea of combining arbitrary Python code with optimized Cython code to work very well. The end result is a much less bloated code base in comparison to something like Java or C++.
Re:Natural Language Processing with Python by yesimahuman on Thursday September 29, 2011 @06:50PM (#3054656)
OpenNLP and Stanford NLP are both Java libraries that might have higher performance characteristics.
Re:Natural Language Processing with Python by jdunck on Thursday September 29, 2011 @01:56PM (#3053493)
I think you should check out PyPy. It has a JIT which significantly improves performance for many use cases.
Re:Natural Language Processing with Python by cgravill on Thursday September 29, 2011 @03:51PM (#3053975)
There are some timing comparisons of using PyPy vs CPython with nltk showing improvements

http://groups.google.com/group/nltk-dev/browse_thread/thread...