AI is killing some companies, yet others are thriving – let's look at the data

Re:AI is killing some companies, yet others are thriving – l... by austin-cheney on Saturday March 01, 2025 @08:11AM (#43218903)

Determine how much you need bots to access your website. If you don't need them then block them. This will kill you if you are reliant upon AD revenue.

On my personal site I started blocking bots and also set a "noindex, nofollow" rule to block web crawlers and search bots too. I noticed no change in traffic and still do about 10,000 visits a month.

[ Parent | on HN ]

Re:AI is killing some companies, yet others are thriving – l... by TZubiri on Saturday March 01, 2025 @11:34AM (#43220948)

This may surprise you, but some bots don't respect robots.txt

[ Parent | on HN ]

Re:AI is killing some companies, yet others are thriving – l... by austin-cheney on Saturday March 01, 2025 @12:10PM (#43221327)

No, but you can identify and block them by other means, such as redirecting bots to an empty page on your web server or blocking them from your router.

[ Parent | on HN ]

Re:AI is killing some companies, yet others are thriving – l... by TZubiri on Saturday March 01, 2025 @12:15PM (#43221367)

but how do you identify them?

[ Parent | on HN ]

Re:AI is killing some companies, yet others are thriving – l... by knowaveragejoe on Monday March 03, 2025 @03:12PM (#43246118)

There's various tricks that can help narrow the criteria, but one that seems particularly good is to put paths no human user would ever find or traverse in robots.txt disallow. Ban anything that visits that path.

[ Parent | on HN ]