News for hackers. Stuff that matters.

A comment on Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines. Parent comment

Re:Show HN: HuggingFace – Fast tokenization library for deep... by nestorD on Monday January 13, 2020 @05:43PM (#22039581)
If I understand you problem clearly, you can use TFIDF to reduce the weight of meaningless words.
[ Parent | on HN ]
Re:Show HN: HuggingFace – Fast tokenization library for deep... by orestis on Monday January 13, 2020 @09:22PM (#22041219)
It’s not meaningless words - it’s common English words that are overloaded and I think considering their position in sentences instead would give better results.

I haven’t yet tried TFIDF though so I’ll see what that will do.

[ Parent | on HN ]