News for hackers. Stuff that matters.

A comment on How Google’s Web Crawler Bypasses Paywalls.

Re:How Google’s Web Crawler Bypasses Paywalls by coverband on Friday February 19, 2016 @02:32PM (#11135941)
My Windows anti-virus deletes the linked sample code automatically upon download, marking it as "Trojan:Win32/Spursint.A". Did anyone have the same experience? (I was actually more interested in using it as a template for writing a simple Chrome extension.)
[ Parent | on HN ]
Re:How Google’s Web Crawler Bypasses Paywalls by mattmaroon on Friday February 19, 2016 @02:57PM (#11136137)
Yep. I then pasted it but it didn't work on wsj.com. Oh well.
[ Parent | on HN ]
Re:How Google’s Web Crawler Bypasses Paywalls by elaineo on Friday February 19, 2016 @03:10PM (#11136240)
try deleting cookies, then hit refresh.
[ Parent | on HN ]
Re:How Google’s Web Crawler Bypasses Paywalls by mrgrieves on Friday February 19, 2016 @05:15PM (#11137079)
Not working here either. I tried wiping out cookies and cache, and Chrome is sending the right user agent (Googlebot).
[ Parent | on HN ]
Re:How Google’s Web Crawler Bypasses Paywalls by nzealand on Saturday February 20, 2016 @01:34AM (#11139179)
Try disabling other extensions that might alter the user agent.
[ Parent | on HN ]