News for hackers. Stuff that matters.

A comment on AI crawlers haven't learned to play nice with websites. Parent comment

Re:AI crawlers haven't learned to play nice with websites by extraduder_ire on Tuesday March 18, 2025 @10:12AM (#43399664)
Did any other crawlers hit it? Would be nice to have someone keeping track of this over time and publishing the data online somewhere.

Probably need to collect logs across multiple sites though, as some crawlers would exclude your domain once they knew.

[ Parent | on HN ]
Re:AI crawlers haven't learned to play nice with websites by blueflow on Tuesday March 18, 2025 @10:15AM (#43399706)
Only one other hit,

2025-03-05 13:32:09 45.89.148.57 (AS46844 SHARKTECH) "Mozilla/5.0 (Windows; U; Windows NT 6.1; pt-BR; rv:1.9.2.18) Gecko/20110614 Firefox/3.6.18 (.NET CLR 3.5.30729)"

[ Parent | on HN ]