Re:Accessing Publicly Available Information on the Internet ...
by alexdoma on Thursday December 14, 2017 @04:28PM
(#15927237)
What would be a better solution, IP address check to allow only known google crawlers perhaps?
Re:Accessing Publicly Available Information on the Internet ...
by andreareina on Thursday December 14, 2017 @07:11PM
(#15928647)
Classify IPs based on their recent behavior[2]. Most bots behave very differently from the median user, along many different dimensions -- volume of requests, time between requests, visit length, which links are followed, etc.
And if this means that bots are altered to become indistinguishable from users, and therefore have a minimal impact on a site's loading? Well, mission accomplished[1].
ETA: [2] Recent behavior (as opposed to all historical behavior) is used so that someone inheriting a "bad" IP isn't completely screwed over.
Re:Accessing Publicly Available Information on the Internet ...
by mark-r on Thursday December 14, 2017 @11:51PM
(#15929730)
That's a superb xkcd that I hadn't seen yet, thanks.
Re:Accessing Publicly Available Information on the Internet ...
by marcosdumay on Thursday December 14, 2017 @06:17PM
(#15928247)
The real solution is disallowing behaviors, instead of shibboleths.
It's surprising that malicious bots aren't exploiting those things already.
Re:Accessing Publicly Available Information on the Internet ...
by ben_w on Friday December 15, 2017 @04:47AM
(#15930705)
That practically invites them to present a different page to google as to a normal user, the former pure SEO, the latter perhaps pure advertising.
Re:Accessing Publicly Available Information on the Internet ...
by fjsolwmv on Friday December 15, 2017 @11:03PM
(#15937542)
And Google will happily deindex the site as soon as they find out
Re:Accessing Publicly Available Information on the Internet ...
by ben_w on Saturday December 16, 2017 @08:55PM
(#15943503)
Raising an interesting question: can a website owner (use the law to) ban google from accessing their website by any mechanism other than their crawler in order that google doesn’t find out?
Sure, obviously limited utility just like “the right to be forgotten”s flaw of diffing USA internet from EU internet to find specifically what people want forgotten, but shenanigans interest me.
Re:Accessing Publicly Available Information on the Internet ...
by someonewithpc on Friday December 15, 2017 @12:33PM
(#15933444)
A related story is that windows 9 isn't a thing because software used to check for windows 95 and 98 by matching the name to "windows 9".
This is an article about the LinkedIn v hiQ case at AdWeek.
It seems AdWeek can distinguish a "good bot" from a "bad bot" irrespective of the behavior of the user^W bot, i.e., whether it is one single HTTP request or 10,000 consecutive requests is irrelevant.How do they do it?
Pattern match against the User-Agent string.
Effective shibboleth.^W engineering.
Clarification: If a user, not a "bot", makes the "wrong" choice of user-agent string (e.g. in the browser settings), then they will be labeled a "bad bot", even if their behavior is no different than other users who are not labeled "bad bots". For example, they make one HTTP GET request just like any other user. There are databases of "acceptable" user-agent strings available to anyone. If still unsure about the point I am making, see this post from several days ago: https://www.sigbus.info/software-compatibility-and-our-own-u...