News for hackers. Stuff that matters.

A comment on Amazon's AI crawler is making my Git server unstable. Parent comment

Re:Amazon's AI crawler is making my Git server unstable by osdotsystem on Tuesday January 21, 2025 @05:27AM (#42778463)
Me i dropped a bunch of dont allows for different bots, as many as i could find.
[ Parent | on HN ]
Re:Amazon's AI crawler is making my Git server unstable by Arnt on Tuesday January 21, 2025 @06:18AM (#42778881)
I'm curious — if that's your attitude, why not all bots? You can say "User-Agent: foo\nDisallow:\n\nUser-Agent: *\nDisallow: /" to allow only named bots and reject them by default.
[ Parent | on HN ]