News for hackers. Stuff that matters.
Re:How Google’s Web Crawler Bypasses Paywalls by mrmcd on Friday February 19, 2016 @01:18PM (#11135313)
Did you read the article. It talks about how that trick no longer works on a lot of sites because they are now checking User-Agent strings too.
Re:How Google’s Web Crawler Bypasses Paywalls by lstamour on Friday February 19, 2016 @01:34PM (#11135448)
Actually I noticed sites have simply changed policies -- if you're a regular visitor your cookies will identify you and block content. The Incognito mode trick works for WSJ and others that would still check the referrer header. Allowing Googlebot access and checking the referrer header are two different things.

Also, Google has published IP addresses it uses, so this extension might not last long...

Re:How Google’s Web Crawler Bypasses Paywalls by slig on Friday February 19, 2016 @01:49PM (#11135587)
> Also, Google has published IP addresses it uses, so this extension might not last long...

They do not [1], but you can find out by doing a reverse DNS query.

> "Google doesn't post a public list of IP addresses for webmasters to whitelist"

[1] https://support.google.com/webmasters/answer/80553?hl=en

Re:How Google’s Web Crawler Bypasses Paywalls by lstamour on Friday February 19, 2016 @06:29PM (#11137577)
Sorry, that was what I meant. :)