>We won’t cover black bots because it is a huge topic with sophisticated analysis and Machine learning algorithms. We will focus on the white and grey bots that identify themselves as such.
This is about not wasting time & effort showing advertising banners to good bots.
I agree with your sentiment, but you should try to be a little more constructive.
Disclaimer: Getting rid of those idiots^wmisguides poor souls is part of my job description.
I don't care about this one person who knows to change the user agent sent, those are the ones you can usually talk to and they'll happily throttle their crawlers.
The other 99 are the problem - and they are a problem that can be solved pretty well by simple string matching.
If I was writing a bot, I would set user agent to some well known and very popular value, i.e. newest Chrome on Windows, or something like that.
The proper way to do this is to block by IP, based on behavior. Block IPs slowing down the site or throw up a captcha like cloudflare does.
Blocking bots sounds great but it just brings Google one step closer to a monopoly. Even good bots just pretend to be people nowadays because lots of people are implementing naive site protection strategies.
Edited: to be less mean