Amazon's AI crawler is making my Git server unstable

Re:Amazon's AI crawler is making my Git server unstable by awsanswers on Saturday January 18, 2025 @02:01PM (#42750525)

Unacceptable, sorry this is happening. Do you know about fail2ban? You can have it automatically filter IPs that violate certain rules. One rule could be matching on the bot trying certain URLs. You might be able to get some kind of honeypot going with that idea. Good luck

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by thayne on Saturday January 18, 2025 @02:12PM (#42750609)

They said that it is coming from different ip addresses every time, so fail2ban wouldn't help.

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by jsheard on Saturday January 18, 2025 @02:24PM (#42750711)

Amazon does publish every IP address range used by AWS, so there is the nuclear option of blocking them all pre-emptively.

https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-rang...

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by xena on Saturday January 18, 2025 @02:27PM (#42750731)

I'd do that, but my DNS is via route 53. Blocking AWS would block my ability to manage DNS automatically as well as certificate issuance via DNS-01.

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by actuallyalys on Saturday January 18, 2025 @02:38PM (#42750793)

They list a service for each address, so maybe you could block all the non-Route 53 IP addresses. Although that assumes they aren’t using the Route 53 IPs or unlisted IPs for scraping (the page warns it’s not a comprehensive list).

Regardless, it sucks that you have to deal with this. The fact that you’re a customer makes it all the more absurd.

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by unsnap_biceps on Saturday January 18, 2025 @02:34PM (#42750771)

If you only block new inbound requests, it shouldn't impact your route 53 or DNS-01 usage.

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by SteveNuts on Saturday January 18, 2025 @02:25PM (#42750712)

It’ll most likely eventually help, as long as they don’t have an infinite address pool.

Do these bots use some client software (browser plugin, desktop app) that’s consuming unsuspecting users bandwidth for distributed crawling?

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by keisborg on Saturday January 18, 2025 @02:24PM (#42750706)

Monitor access logs for links that only crawlers can find.

Edit: oh, I got your point now.

[ Parent | on HN ]