Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on AI bots everywhere. Does anyone have a good whitelist for robots.txt?.
And the user-agent sent is completely in control of the bad actor. They can send their user-agent as "google bot".
You would need something like WAF from https://www.cloudflare.com/ or AWS
There are no comments on this comment yet.
And the user-agent sent is completely in control of the bad actor. They can send their user-agent as "google bot".
You would need something like WAF from https://www.cloudflare.com/ or AWS