News for hackers. Stuff that matters.

A comment on Llms.txt. Parent comment

Re:Llms.txt by Tepix on Wednesday September 04, 2024 @05:17AM (#41443541)
Ideally i would like these crawlers to access /robots.txt but nothing else.

Only if they ignore robots.txt the access rules will stop them.

[ Parent | on HN ]
Re:Llms.txt by jraph on Wednesday September 04, 2024 @07:06AM (#41444229)
You can probably write a specific location block for robots.txt which will have a higher priority.

See also https://stackoverflow.com/questions/5238377/nginx-location-p...

[ Parent | on HN ]