Re:How Google’s Web Crawler Bypasses Paywalls
by swanson on Friday February 19, 2016 @02:17PM
(#11135829)
I wonder if a user-agent that was something like "Not a Googlebot" would a) allow access (probably regex based) and b) be truthful/plausible deniability.
[ Parent
| on HN ]
Re:How Google’s Web Crawler Bypasses Paywalls
by oneeyedpigeon on Friday February 19, 2016 @05:43PM
(#11137292)
What is your purpose in setting your user-agent string to that value, other than cleverly bypassing the paywall?