Re:How to detect web bots?
by hinkley on Sunday February 09, 2020 @01:25PM
(#22283483)
I joke that I only have two real skills. Asking questions and identifying good ideas.
In the latter category I have heterogenous load balancing, segregated by traffic type. For example, if you have a read-mostly web application, often the most expensive parts are in the administrative functions. If they run on separate hardware, then a regression doesn’t affect everyone, and a client side bug or flood of traffic doesn’t automatically lock people out of the admin workflow.
Similarly, you have server side rendering for web crawlers and anyone with JavaScript disabled for Angular and other interactive frameworks, which generally involves running a small quantity of specialized services on a few machines.
Shunting all users who smell like a bot to a small set of machines doesn’t lock them out, but it does preserve the experience for everybody else.
In the latter category I have heterogenous load balancing, segregated by traffic type. For example, if you have a read-mostly web application, often the most expensive parts are in the administrative functions. If they run on separate hardware, then a regression doesn’t affect everyone, and a client side bug or flood of traffic doesn’t automatically lock people out of the admin workflow.
Similarly, you have server side rendering for web crawlers and anyone with JavaScript disabled for Angular and other interactive frameworks, which generally involves running a small quantity of specialized services on a few machines.
Shunting all users who smell like a bot to a small set of machines doesn’t lock them out, but it does preserve the experience for everybody else.