Re:The Smart TV in Your LivingRoom Is a Node in the AIScrapi...
by blakesterz on Saturday June 06, 2026 @09:32AM
(#48424946)
Are there any defenses I can put in front of my websites that are good for stopping these things? The amount of traffic I see from residential proxies is just killing me. In particular defense against residential proxies.
[ Parent
| on HN ]
Re:The Smart TV in Your LivingRoom Is a Node in the AIScrapi...
by jappgar on Saturday June 06, 2026 @10:49AM
(#48425647)
The bots used by these proxies are detectable in a few ways. Remember the bot itself doesn't run on the proxy...
There is discernible lag from proxy to c&c node. The individual bots don't have access to a lot of compute, and are sometimes restricted wrt feature set (e.g. proprietary video codecs).
There are a few other techniques. It's a cat and mouse game though. And the bot owners are usually more motivated than you are.
Re:The Smart TV in Your LivingRoom Is a Node in the AIScrapi...
by bakugo on Saturday June 06, 2026 @11:12AM
(#48425834)
Add a captcha or proof-of-work challenge in front of your website. Those are pretty much your only options.
Re:The Smart TV in Your LivingRoom Is a Node in the AIScrapi...
by asdff on Saturday June 06, 2026 @02:29PM
(#48427594)
There's probably ways to go on the offensive too with pages that might be hidden from human users but still crawled by bots.
Re:The Smart TV in Your LivingRoom Is a Node in the AIScrapi...
by trumpdong on Saturday June 06, 2026 @12:54PM
(#48426762)
Make your server so efficient that a few extra requests doesn't bring it down.
Alternatively, if it's the first time the IP is seen and it's a deep linked page with no referer, send a neverending chunked gzip data stream.
Re:The Smart TV in Your LivingRoom Is a Node in the AIScrapi...
by asdff on Saturday June 06, 2026 @02:28PM
(#48427586)
Have a link on your page that would be hidden by users via css. e.g. white text on white background. Have it just abuse compute or do something absolutely stupid for the bots that end up crawling over that link. Hell, just zip bomb them.
Re:The Smart TV in Your LivingRoom Is a Node in the AIScrapi...
by Rasbora on Saturday June 06, 2026 @06:17PM
(#48429567)
Solutions exist specifically for detecting residential proxy traffic: https://layer3intel.com/tripwire
You can also check if your network is running a residential proxy exit node here: https://layer3intel.com/is-my-network-a-residential-proxy