News for hackers. Stuff that matters.
Re:Ask HN: Are there any purely keyword-based search engines... by jedberg on Sunday March 25, 2018 @12:16PM (#16672832)
The crawl is never done. It's constantly crawling. And you're right, it's just a queue of websites that's been found, and when a crawling machine has an available slot, it crawls the next site (obviously that's simplified).

Certain pages get crawled more often than others. As a website owner, you can tell Google how often your content changes, which they will use as a clue for how often to recrawl your site.

If you're really big (like for example Reddit), you actually lose that control in Google dashboard -- they fully control the crawl rate. From experience, I can tell you that they are crawling large sites, like Reddit, continuously. Even 7 years ago, when I last worked there, they were crawling Reddit so much that we had to set up a separate server infrastructure just to respond to Google's requests, because their access patterns were so different from every other user.

Re:Ask HN: Are there any purely keyword-based search engines... by owenversteeg on Sunday March 25, 2018 @07:23PM (#16675256)
That's fascinating. Is there any way to contact Google to get them to change their crawl patterns? Or a more efficient way to directly send them the site updates as they happen? I imagine a "direct line" to Google would save resources both on Google's side and yours, but I've never ran a site big enough for that to be a problem so I wouldn't know.
Re:Ask HN: Are there any purely keyword-based search engines... by himlion on Monday March 26, 2018 @08:18AM (#16678253)
I think that would lend itself to abuse.