Llms.txt - Hackdot

Re:Llms.txt by jraph on Wednesday September 04, 2024 @01:54AM (#41442320)

I've seen some of these bots take a lot of CPU on my server, especially when browsing my (very small) forgejo instance. I banned them with a 444 error [1] in the reverse proxy settings as a temporary measure that became permanent, and then some more from this list [2], but I will consider yours as well, thanks for sharing.

    if ($http_user_agent ~ facebook) { return 444; }
    if ($http_user_agent ~ Amazonbot) { return 444; }
    if ($http_user_agent ~ Bytespider) { return 444; }
    if ($http_user_agent ~ GPTBot) { return 444; }
    if ($http_user_agent ~ ClaudeBot) { return 444; }
    if ($http_user_agent ~ ImagesiftBot) { return 444; }
    if ($http_user_agent ~ CCBot) { return 444; }
    if ($http_user_agent ~ ChatGPT-User) { return 444; }
    if ($http_user_agent ~ omgili) { return 444; }
    if ($http_user_agent ~ Diffbot) { return 444; }
    if ($http_user_agent ~ Claude-Web) { return 444; }
    if ($http_user_agent ~ PerplexityBot) { return 444; }

(edit: see replies to do it in a cleaner way)

[1] https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#ngin...

[2] https://blog.cloudflare.com/declaring-your-aindependence-blo...

[ Parent | on HN ]

Re:Llms.txt by otherme123 on Wednesday September 04, 2024 @02:25AM (#41442479)

In your nginx.conf, http block, add

    include /etc/nginx/useragent.rules;

In /etc/nginx/useragent.rules

    map $http_user_agent $badagent {
        default 0;
        ~facebook 1;
        [...]
        ~PerplexityBot 1;
    }

In your site.conf, server block, add

    if ($badagent) {
        return 444;
    }

[ Parent | on HN ]

Re:Llms.txt by nilsherzig on Wednesday September 04, 2024 @04:15AM (#41443123)

Anyone knows of a crowd sourced list of these user agents? With the current state of AI startups it will be hard to keep this up to date by myself

[ Parent | on HN ]

Re:Llms.txt by jraph on Wednesday September 04, 2024 @03:01AM (#41442662)

Ah, nice, way better than my string of ifs.

[ Parent | on HN ]

Re:Llms.txt by Tepix on Wednesday September 04, 2024 @05:17AM (#41443541)

Ideally i would like these crawlers to access /robots.txt but nothing else.

Only if they ignore robots.txt the access rules will stop them.

[ Parent | on HN ]

Re:Llms.txt by jraph on Wednesday September 04, 2024 @07:06AM (#41444229)

You can probably write a specific location block for robots.txt which will have a higher priority.

[ Parent | on HN ]