News for hackers. Stuff that matters.
Re:How Google’s Web Crawler Bypasses Paywalls by elaineo on Friday February 19, 2016 @01:25PM (#11135387)
Google is not okay with cloaking, but they will whitelist publishers if the publisher specifically includes a parameter that declares if the site requires registration or subscription. This is done in the sitemap.

https://support.google.com/news/publisher/answer/74288?hl=en

Re:How Google’s Web Crawler Bypasses Paywalls by ikeboy on Friday February 19, 2016 @02:25PM (#11135881)
https://support.google.com/news/publisher/answer/40543?hl=en seems to specifically ban this. WSJ is in violation, not fitting any of the categories there.
Re:How Google’s Web Crawler Bypasses Paywalls by slig on Friday February 19, 2016 @01:29PM (#11135417)
I didn't know that, thanks. But reading, it seems that option is about Google News, not the main Google Search.

> News-specific tag definitions

> Yes, if access is not open, else should be omitted

> Possible values include "Subscription" or "Registration", describing the accessibility of the article. If the article is accessible to Google News readers without a registration or subscription, this tag should be omitted.

Re:How Google’s Web Crawler Bypasses Paywalls by pacquiao882 on Friday February 19, 2016 @03:58PM (#11136566)
WSJ is big enough to negotiate their own terms with Google Search.
Re:How Google’s Web Crawler Bypasses Paywalls by morgante on Friday February 19, 2016 @07:27PM (#11137854)
They're really not. They need Google a lot more than Google needs them.

One of these things is going to happen:

(1) They end this "experiment."

(2) They stop serving Google the full content. (And see their rankings drop accordingly.)

(3) They get delisted for cloaking.