News for hackers. Stuff that matters.

A comment on Llms.txt. Parent comment

Re:Llms.txt by otherme123 on Wednesday September 04, 2024 @02:25AM (#41442479)
In your nginx.conf, http block, add

    include /etc/nginx/useragent.rules;
In /etc/nginx/useragent.rules

    map $http_user_agent $badagent {
        default 0;
        ~facebook 1;
        [...]
        ~PerplexityBot 1;
    }
In your site.conf, server block, add

    if ($badagent) {
        return 444;
    }
[ Parent | on HN ]
Re:Llms.txt by nilsherzig on Wednesday September 04, 2024 @04:15AM (#41443123)
Anyone knows of a crowd sourced list of these user agents? With the current state of AI startups it will be hard to keep this up to date by myself
[ Parent | on HN ]
Re:Llms.txt by jraph on Wednesday September 04, 2024 @03:01AM (#41442662)
Ah, nice, way better than my string of ifs.
[ Parent | on HN ]
Re:Llms.txt by Tepix on Wednesday September 04, 2024 @05:17AM (#41443541)
Ideally i would like these crawlers to access /robots.txt but nothing else.

Only if they ignore robots.txt the access rules will stop them.

[ Parent | on HN ]
Re:Llms.txt by jraph on Wednesday September 04, 2024 @07:06AM (#41444229)
You can probably write a specific location block for robots.txt which will have a higher priority.

See also https://stackoverflow.com/questions/5238377/nginx-location-p...

[ Parent | on HN ]