Re:How Google’s Web Crawler Bypasses Paywalls
by elaineo on Friday February 19, 2016 @01:25PM
(#11135387)
Google is not okay with cloaking, but they will whitelist publishers if the publisher specifically includes a parameter that declares if the site requires registration or subscription. This is done in the sitemap.
https://support.google.com/news/publisher/answer/74288?hl=en
Re:How Google’s Web Crawler Bypasses Paywalls
by ikeboy on Friday February 19, 2016 @02:25PM
(#11135881)
https://support.google.com/news/publisher/answer/40543?hl=en seems to specifically ban this. WSJ is in violation, not fitting any of the categories there.
Re:How Google’s Web Crawler Bypasses Paywalls
by slig on Friday February 19, 2016 @01:29PM
(#11135417)
I didn't know that, thanks. But reading, it seems that option is about Google News, not the main Google Search.
> News-specific tag definitions
> Yes, if access is not open, else should be omitted
> Possible values include "Subscription" or "Registration", describing the accessibility of the article. If the article is accessible to Google News readers without a registration or subscription, this tag should be omitted.
Re:How Google’s Web Crawler Bypasses Paywalls
by pacquiao882 on Friday February 19, 2016 @03:58PM
(#11136566)
WSJ is big enough to negotiate their own terms with Google Search.
Re:How Google’s Web Crawler Bypasses Paywalls
by morgante on Friday February 19, 2016 @07:27PM
(#11137854)
They're really not. They need Google a lot more than Google needs them.
One of these things is going to happen:
(1) They end this "experiment."
(2) They stop serving Google the full content. (And see their rankings drop accordingly.)
(3) They get delisted for cloaking.
Re:How Google’s Web Crawler Bypasses Paywalls
by CM30 on Friday February 19, 2016 @04:07PM
(#11136630)
Yes, and I suspect that's what going to get them to change it back again. After a ban from Google and the traffic drop it'll likely bring, that paywall is likely coming right back down. An awful lot of media companies made similar mistakes before, and it's always ended with them quickly removing their 'work arounds'.
Re:How Google’s Web Crawler Bypasses Paywalls
by Implicated on Friday February 19, 2016 @07:35PM
(#11137917)
Yea, except they're not going to get that ban.
[1]: https://support.google.com/webmasters/answer/66355?hl=en