Amazon's AI crawler is making my Git server unstable

Re:Amazon's AI crawler is making my Git server unstable by xena on Saturday January 18, 2025 @02:27PM (#42750731)

I'd do that, but my DNS is via route 53. Blocking AWS would block my ability to manage DNS automatically as well as certificate issuance via DNS-01.

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by actuallyalys on Saturday January 18, 2025 @02:38PM (#42750793)

They list a service for each address, so maybe you could block all the non-Route 53 IP addresses. Although that assumes they aren’t using the Route 53 IPs or unlisted IPs for scraping (the page warns it’s not a comprehensive list).

Regardless, it sucks that you have to deal with this. The fact that you’re a customer makes it all the more absurd.

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by unsnap_biceps on Saturday January 18, 2025 @02:34PM (#42750771)

If you only block new inbound requests, it shouldn't impact your route 53 or DNS-01 usage.

[ Parent | on HN ]