News for hackers. Stuff that matters.
Re:A First Excercise in Natural Language Processing with Pyt... by dec0dedab0de on Friday September 08, 2017 @07:34PM (#15204747)
I get that the point is to be an introduction to the libraries and whatnot, but was I the only one who immediately thought of just using Counter?

    from collections import Counter
    import re

    [word for word, count in Counter(re.findall('\w*', text.lower())).items() if count == 1]
Re:A First Excercise in Natural Language Processing with Pyt... by kleiba on Saturday September 09, 2017 @04:24AM (#15206084)
As the article states in the introductory paragraph, this problem encompasses more than just counting strings. It also involves "some fundamental tasks of natural language processing (NLP): tokenization (dividing a text into words), stemming, and part-of-speech tagging for lemmatization", so a little more work is required here.
Re:A First Excercise in Natural Language Processing with Pyt... by cristoperb on Friday September 08, 2017 @09:10PM (#15205055)
Yep, if you don't need any of the fancy NLP features of a library then something like this is the most straight-forward. (In the article I did give a plain Python solution using split() to tokenize and then using Counter to get the hapaxes in the function called "word_form_hapaxes".)
Re:A First Excercise in Natural Language Processing with Pyt... by newman8r on Friday September 08, 2017 @09:13PM (#15205064)
for anyone interested in more good beginner resources, I really enjoyed this youtube playlist on python NLTK https://www.youtube.com/watch?v=OGxgnH8y2NM&list=PLQVvvaa0Qu...

edit* I accidentally linked to another good playlist, but here's the first vid of the NLTK list from the same user https://www.youtube.com/watch?v=FLZvOKSCkxY

Re:A First Excercise in Natural Language Processing with Pyt... by grabcocque on Friday September 08, 2017 @06:45PM (#15204499)
Hapax Legomenon is such a satisfying phrase to say. Even the opportunity to look at it makes my eyes happy.
Re:A First Excercise in Natural Language Processing with Pyt... by visarga on Saturday September 09, 2017 @03:13AM (#15205923)
I counted word n-grams up to length 6 in a corpus of 6 billion words with Madoka, a Count-Min sketch algorithm.

https://pypi.python.org/pypi/madoka

Re:A First Excercise in Natural Language Processing with Pyt... by cristoperb on Friday September 08, 2017 @09:34PM (#15205134)
Author here. The misspelling in the title is embarrassing, but luckily not very noticeable (I've fixed it on the site).
Re:A First Excercise in Natural Language Processing with Pyt... by bluzeee on Saturday September 09, 2017 @03:53AM (#15206003)
It's so beautifully put together and made so easy to understand, I thank you very much, it helped greately in my learning.