News for hackers. Stuff that matters.

A comment on Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines. Parent comment

Re:Show HN: HuggingFace – Fast tokenization library for deep... by orestis on Monday January 13, 2020 @02:56PM (#22037960)
It might be that the corpus I was trying to cluster needs better preprocessing, or perhaps better n-grams. Using Bigrams only I saw a lot of common words that were meaningless, but adding them as stop words made the results worse. Hence my wondering if some other vectorization would produce better results.

On a related note, as a newcomer just trying to get things done (i.e. applied NLP) I find the whole ecosystem great but frustrating, so many frameworks and libraries but not clear ways to compose them together. Any resources out there that help make a sense of things?

[ Parent | on HN ]
Re:Show HN: HuggingFace – Fast tokenization library for deep... by nestorD on Monday January 13, 2020 @05:43PM (#22039581)
If I understand you problem clearly, you can use TFIDF to reduce the weight of meaningless words.
[ Parent | on HN ]
Re:Show HN: HuggingFace – Fast tokenization library for deep... by orestis on Monday January 13, 2020 @09:22PM (#22041219)
It’s not meaningless words - it’s common English words that are overloaded and I think considering their position in sentences instead would give better results.

I haven’t yet tried TFIDF though so I’ll see what that will do.

[ Parent | on HN ]