Re:How Google’s Web Crawler Bypasses Paywalls
by dheera on Friday February 19, 2016 @02:55PM
(#11136127)
In that case, I'm just going to set my User-Agent permanently to Google's crawler for fun and see how the web renders, in general. Just for fun, because I want to see what the world looks like from a different perspective. That's my intent. I've stated it. Now that my user agent has been changed, I'm going to go grab lunch, carry on with life, maybe check out some cat pictures and then maybe some news sites over tea and snacks, and then go for a run.
[ Parent
| on HN ]
Re:How Google’s Web Crawler Bypasses Paywalls
by lojack on Friday February 19, 2016 @03:39PM
(#11136448)
In my opinion the WSJ has a major rendering bug that affects everyone except for the google search crawler, and I selectively set my User Agent to get around said bug.
Re:How Google’s Web Crawler Bypasses Paywalls
by comex on Friday February 19, 2016 @04:03PM
(#11136607)
In any judge's opinion, you'd be full of shit. Judges aren't stupid.
Which doesn't matter, because the WSJ is never going to sue you. But make sure you only consider your justification a personal one, not one that would provide any legal protection.
Re:How Google’s Web Crawler Bypasses Paywalls
by djsumdog on Friday February 19, 2016 @05:10PM
(#11137053)
Most judges and legislators are fucking morons. Legal systems aren't a science. I wrote a thing on it a few years ago:
Re:How Google’s Web Crawler Bypasses Paywalls
by newjersey on Friday February 19, 2016 @05:29PM
(#11137198)
I thought the problem was that it doesn't matter if WSJ is malicious in this case. Because cfaa is a criminal matter, zealous police departments and district attorney's offices can pursue heavy handed cases even without much cooperation from "the victim" which in this case is a corporation.
All I want to say is scrape cfaa now!