Re:Two Upstart Search Engines Are Teaming Up to Take on Google
by 1024core on Tuesday November 12, 2024 @04:10PM
(#42119744)
Is Common Crawl updated frequently?
Re:Two Upstart Search Engines Are Teaming Up to Take on Google
by astonex on Tuesday November 12, 2024 @05:13PM
(#42120353)
They do monthly-ish releases https://index.commoncrawl.org/
Are any individual users downloading CC for potential use in the future?
It may seem like a non-trivial task to process ~100TB at home today but in the future processing this amount of data will likely seem trivial. CC data is available for download to anyone but, to me, it appears only so-called "tech" companies and "researchers" are grabbing a copy.
Many years ago I began storing copies of the publicly-available com. and net. zonefiles from Verisign. At the time it was infeasible for me to try to serve multi-GB zonefiles on the local network at home. Today, it's feasible. And in the future it will be even easier.
NB. I am not employed by a so-called "tech" company. I store this data for personal, non-commercial use.