Amazon's AI crawler is making my Git server unstable

Re:Amazon's AI crawler is making my Git server unstable by osdotsystem on Saturday January 18, 2025 @04:20AM (#42746992)

I feel for you. Same for OpenAI (https://www.linkedin.com/posts/appinv_openai-is-a-felon-comp...). A friend told me Ai bots even ignored the robots.tx ...

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by Arnt on Saturday January 18, 2025 @11:15AM (#42749362)

I've heard that too. Your posting inspired me to check. I see daily accesses from claudebot and gtpbot to /robots.txt on two sites. Also from something called amazonbot, which may be another AI bot?

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by osdotsystem on Tuesday January 21, 2025 @05:27AM (#42778463)

Me i dropped a bunch of dont allows for different bots, as many as i could find.

[ Parent | on HN ]

Re:Amazon's AI crawler is making my Git server unstable by Arnt on Tuesday January 21, 2025 @06:18AM (#42778881)

I'm curious — if that's your attitude, why not all bots? You can say "User-Agent: foo\nDisallow:\n\nUser-Agent: *\nDisallow: /" to allow only named bots and reject them by default.

[ Parent | on HN ]