A First Excercise in Natural Language Processing with Python: Counting Hapaxes

Re:A First Excercise in Natural Language Processing with Pyt... by dec0dedab0de on Friday September 08, 2017 @07:34PM (#15204747)

I get that the point is to be an introduction to the libraries and whatnot, but was I the only one who immediately thought of just using Counter?

    from collections import Counter
    import re

    [word for word, count in Counter(re.findall('\w*', text.lower())).items() if count == 1]

[ Parent | on HN ]

Re:A First Excercise in Natural Language Processing with Pyt... by kleiba on Saturday September 09, 2017 @04:24AM (#15206084)

As the article states in the introductory paragraph, this problem encompasses more than just counting strings. It also involves "some fundamental tasks of natural language processing (NLP): tokenization (dividing a text into words), stemming, and part-of-speech tagging for lemmatization", so a little more work is required here.

[ Parent | on HN ]

Re:A First Excercise in Natural Language Processing with Pyt... by cristoperb on Friday September 08, 2017 @09:10PM (#15205055)

Yep, if you don't need any of the fancy NLP features of a library then something like this is the most straight-forward. (In the article I did give a plain Python solution using split() to tokenize and then using Counter to get the hapaxes in the function called "word_form_hapaxes".)

[ Parent | on HN ]