News for hackers. Stuff that matters.
Re:How Google’s Web Crawler Bypasses Paywalls by swanson on Friday February 19, 2016 @02:17PM (#11135829)
I wonder if a user-agent that was something like "Not a Googlebot" would a) allow access (probably regex based) and b) be truthful/plausible deniability.
Re:How Google’s Web Crawler Bypasses Paywalls by oneeyedpigeon on Friday February 19, 2016 @05:43PM (#11137292)
What is your purpose in setting your user-agent string to that value, other than cleverly bypassing the paywall?