News for hackers. Stuff that matters.

A comment on Building a Search Engine from Scratch.

Re:Building a Search Engine from Scratch by sterlind on Friday December 06, 2019 @02:35PM (#21725207)
Do you use Common Crawl? It seems like a pretty big corpus, reasonably up-to-date and free. It seems like a good way to supplement the page data from the browser extension.
[ Parent | on HN ]
Re:Building a Search Engine from Scratch by sammacbeth_ on Friday December 06, 2019 @02:44PM (#21725284)
From the article:

"It may seem like Common Crawl would suffice for this purpose, but it has poor coverage outside of the US and its update frequency is not realistic for use in a search engine."

[ Parent | on HN ]