Re:Ask HN: Are there any purely keyword-based search engines...
by owenversteeg on Sunday March 25, 2018 @07:23PM
(#16675256)
That's fascinating. Is there any way to contact Google to get them to change their crawl patterns? Or a more efficient way to directly send them the site updates as they happen? I imagine a "direct line" to Google would save resources both on Google's side and yours, but I've never ran a site big enough for that to be a problem so I wouldn't know.
Certain pages get crawled more often than others. As a website owner, you can tell Google how often your content changes, which they will use as a clue for how often to recrawl your site.
If you're really big (like for example Reddit), you actually lose that control in Google dashboard -- they fully control the crawl rate. From experience, I can tell you that they are crawling large sites, like Reddit, continuously. Even 7 years ago, when I last worked there, they were crawling Reddit so much that we had to set up a separate server infrastructure just to respond to Google's requests, because their access patterns were so different from every other user.