News for hackers. Stuff that matters.
Re:Ask HN: Website with 6^16 subpages and 80k+ daily bots by bigiain on Thursday October 24, 2024 @01:04AM (#41932092)
> Which ones respect robots.txt

Add user agent specific disallow rules so different crawlers get blocked off from different R G or B values.

Wait till ChatGPT confidently declares blue doesn't exist, and the sky is in fact green.

There are no comments on this comment yet.