thomvaughan

It's worth noting that building a solid search index needs more than just text. The full Common Crawl archive includes metadata, links, page structure, and other signals that are needed for relevance and ranking, and to date is over 7 PiB, so that's rather more than tends to fit on home servers