Re:Two Upstart Search Engines Are Teaming Up to Take on Google
by stephantul on Tuesday November 12, 2024 @10:06AM
(#42115997)
I meant this in response to the parent that Common Crawl only updates every month, which seemed to imply that this was sufficient.
This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.
This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.