Re:Accessing Publicly Available Information on the Internet ...
by andreareina on Thursday December 14, 2017 @07:11PM
(#15928647)
Classify IPs based on their recent behavior[2]. Most bots behave very differently from the median user, along many different dimensions -- volume of requests, time between requests, visit length, which links are followed, etc.
And if this means that bots are altered to become indistinguishable from users, and therefore have a minimal impact on a site's loading? Well, mission accomplished[1].
And if this means that bots are altered to become indistinguishable from users, and therefore have a minimal impact on a site's loading? Well, mission accomplished[1].
[1] https://xkcd.com/810/
ETA: [2] Recent behavior (as opposed to all historical behavior) is used so that someone inheriting a "bad" IP isn't completely screwed over.