Blocking LLM crawlers, without JavaScript

Re:Blocking LLM crawlers, without JavaScript by bakigul on Sunday November 16, 2025 @05:59AM (#45944142)

Why do people want to block LLM crawlers? Everyone wants to be visible in GPT, Claude, and others, don’t they?

[ Parent | on HN ]

Re:Blocking LLM crawlers, without JavaScript by saltysalt on Sunday November 16, 2025 @06:18AM (#45944222)

Exactly! With many people moving to AI-driven search, if you are not in their indexes you traffic will reduce even further.

And I say this as someone who built a search engine with no AI: I know my audience for that service is very niche, with the vast majority of people using AI search because it's more convenient.

[ Parent | on HN ]

Re:Blocking LLM crawlers, without JavaScript by suckler on Sunday November 16, 2025 @06:31AM (#45944288)

You're a funny guy.

[ Parent | on HN ]

Re:Blocking LLM crawlers, without JavaScript by ThomasMidgley on Sunday November 16, 2025 @07:03AM (#45944466)

[ Parent | on HN ]

Re:Blocking LLM crawlers, without JavaScript by aiven on Sunday November 16, 2025 @06:13AM (#45944209)

- If LLM knows about your content, people don't really need to visit your site - LLM crawlers can be pretty aggressive and eat up a lot of traffic - Google will not suggest its misleading "summaries" from its search for your site - Some people just hate LLMs that much ¯\_(ツ)_/¯

[ Parent | on HN ]

Re:Blocking LLM crawlers, without JavaScript by bo1024 on Sunday November 16, 2025 @11:27AM (#45946287)

One issue is just the sheer amount of such traffic.

[ Parent | on HN ]

Re:Blocking LLM crawlers, without JavaScript by xgulfie on Sunday November 16, 2025 @01:47PM (#45947378)

If someone doesn't want to visit my website then why would I provide my website to them

[ Parent | on HN ]