News for hackers. Stuff that matters.

A comment on Bypassing website anti-scraping protections. Parent comment

Re:Bypassing website anti-scraping protections by matheusmoreira on Friday June 01, 2018 @03:10PM (#17208909)
>Many paywalled sites will let googlebot index their content but block other crawlers.

Doesn't that infringe upon Google's own rules? I always thought Google didn't like it when sites served its crawler content that's different from what users get when they follow Google's link.

[ Parent | on HN ]
Re:Bypassing website anti-scraping protections by fauigerzigerk on Friday June 01, 2018 @03:24PM (#17209058)
That's why many paywalled sites give you a few free articles per month if you're coming from a Google search results page.

But it no longer works at all sites. Maybe the rule has been dropped now that paywalls are becoming more popular (with publishers that is)

[ Parent | on HN ]