Re:Common Crawl
by th0ma5 on Friday March 26, 2021 @03:38PM
(#26596054)
Actually I was able to search URLs with very little ram across the entire collection as they have a series of indexes you can download.
In theory someone could do something similar with terms, or you could first use URLs to filter the text size you download into Elastic or Solr and do your own custom search that way.
The indexes are really neat though, I highly recommend playing with them.
In theory someone could do something similar with terms, or you could first use URLs to filter the text size you download into Elastic or Solr and do your own custom search that way.
The indexes are really neat though, I highly recommend playing with them.