Making the Internet Archive’s full text search faster

Re:Making the Internet Archive’s full text search faster by mdellabitta on Saturday June 10, 2017 @10:59AM (#14527572)

Hey, I'm wondering why you didn't consider using stopwords to prevent bloated inverted index entries fir words like 'the'?

[ Parent | on HN ]

Re:Making the Internet Archive’s full text search faster by gioark on Saturday June 10, 2017 @11:25AM (#14527668)

We don't use stopwords because we want to find all the best and complete matches. We don't want to ignore any of the words part of the search query.

[ Parent | on HN ]

Re:Making the Internet Archive’s full text search faster by aisofteng on Saturday June 10, 2017 @12:05PM (#14527816)

You do use stopwords. Your most common unigrams are not in the index, by design. You just use your own stopwords.

[ Parent | on HN ]