News for hackers. Stuff that matters.

gioark

Re:Making the Internet Archive’s full text search faster on Saturday June 10, 2017 @11:28AM (#14527680)
sure, feel free to contact me by email or DM on twitter. gio archive org
Re:Making the Internet Archive’s full text search faster on Saturday June 10, 2017 @11:25AM (#14527668)
We don't use stopwords because we want to find all the best and complete matches. We don't want to ignore any of the words part of the search query.
hi, I am the author of the article.

@devhead: To import the data into ES we used a custom application to extract the text from the OCR'd documents. This is required to support our bookreader software. A complete ingestion takes a few days; we rate-limit indexing in order not to overload the cluster, and maintain reasonable search performance.

If you are having problems loading the page, try this url: https://medium.com/@giovannidamiola/making-the-internet-arch...