Re:Making the Internet Archive’s full text search faster
by mdellabitta on Saturday June 10, 2017 @10:59AM
(#14527572)
Hey, I'm wondering why you didn't consider using stopwords to prevent bloated inverted index entries fir words like 'the'?
Re:Making the Internet Archive’s full text search faster
by gioark on Saturday June 10, 2017 @11:25AM
(#14527668)
We don't use stopwords because we want to find all the best and complete matches. We don't want to ignore any of the words part of the search query.
Re:Making the Internet Archive’s full text search faster
by aisofteng on Saturday June 10, 2017 @12:05PM
(#14527816)
You do use stopwords. Your most common unigrams are not in the index, by design. You just use your own stopwords.
Re:Making the Internet Archive’s full text search faster
by kampsy on Saturday June 10, 2017 @05:03AM
(#14526683)
Great article man. Am always super exited when i find articles that talk about information retrieval systems. I have a lot of questions for you. Been working on a search engine project www.cognifly.com for a year and its inverted index is still very small. Like 4gig now. So is ok if i send you an email?
Re:Making the Internet Archive’s full text search faster
by gioark on Saturday June 10, 2017 @11:28AM
(#14527680)
sure, feel free to contact me by email or DM on twitter.
gio archive org
Re:Making the Internet Archive’s full text search faster
by giodamelio on Friday June 09, 2017 @10:03PM
(#14525806)
It was a great article. On a side note, holy shit is it rare to even hear of other guys named Giovanni, much less as similar a last name (I'm Giovanni d'Amelio).
@devhead: To import the data into ES we used a custom application to extract the text from the OCR'd documents. This is required to support our bookreader software. A complete ingestion takes a few days; we rate-limit indexing in order not to overload the cluster, and maintain reasonable search performance.