News for hackers. Stuff that matters.
Re:Web Scraping in 2016 by Guest98123 on Tuesday August 23, 2016 @09:36PM (#12349068)
Google usually does 300 pages a minute on my site. In total, bots were loading about 1,000 pages a minute.
Re:Web Scraping in 2016 by encoderer on Tuesday August 23, 2016 @09:45PM (#12349105)
That is a lot! It's also still an order of magnitude less than big content sites. Not taking anything away from what must be a successful website to get a consistent 300 pages/minute crawl rate, but only to illustrate magnitude.
Re:Web Scraping in 2016 by Guest98123 on Tuesday August 23, 2016 @10:08PM (#12349185)
I was curious, so I just checked the stats through webmaster tools. For the last 90 days, the low is 450,000 daily crawled pages, average is 650,000, and yesterday was the high of 1,130,000 (780 per minute). Ouch.
Re:Web Scraping in 2016 by encoderer on Tuesday August 23, 2016 @10:16PM (#12349206)
Have you seen correlation between rankings and crawl rate?
Re:Web Scraping in 2016 by Guest98123 on Tuesday August 23, 2016 @10:42PM (#12349322)
This particular site is top 5,000 Alexa. The content changes every minute, and Google is fast at picking up those changes. The last cache of the homepage was 7 minutes ago from Google.

There's definitely a correlation between my sites' Google rankings, their organic traffic, and their crawl rate. The other sites I run are Alexa top 30,000 and top 100,000. They all feature dynamically changing content, but Google is definitely using a higher crawl rate on my higher ranking sites. This isn't a surprise though, Google has limited resources like everyone, and they'll focus those resources in a way that provides the most benefit.

Edit: If you're talking about the correlation between daily ranking and daily crawl rate for an individual site, then no, I'm not aware of any patterns. For example, the graph is flat for organic traffic and total indexed pages, but the crawl rate jumps up and down as mentioned, and it doesn't appear to relate on a daily basis.

Re:Web Scraping in 2016 by encoderer on Tuesday August 23, 2016 @11:49PM (#12349543)
I've seen rankings drops following drops in crawl rate.