News for hackers. Stuff that matters.

A comment on Yep: Google alternative that shares revenue with creators – by Ahrefs. Parent comment

Re:Yep: Google alternative that shares revenue with creators... by dazc on Saturday June 04, 2022 @02:31AM (#31617622)
What's stopping you, they crawl your site day and night and sell the data they have harvested at a vast profit. They don't even respect robots.txt.
[ Parent | on HN ]
Re:Yep: Google alternative that shares revenue with creators... by botsbreeder on Saturday June 04, 2022 @02:55AM (#31617731)
AhrefsBot respects robots.txt. Details are here: https://ahrefs.com/robot
[ Parent | on HN ]
Re:Yep: Google alternative that shares revenue with creators... by blitzar on Saturday June 04, 2022 @06:16AM (#31618727)
My parents say I am a very good boy too.
[ Parent | on HN ]
Re:Yep: Google alternative that shares revenue with creators... by agmater on Saturday June 04, 2022 @11:50AM (#31621783)
If you don't believe the CEO you can always just IP block them. In my experience they respect "User-agent: * Disallow: /".
[ Parent | on HN ]
Re:Yep: Google alternative that shares revenue with creators... by dazc on Saturday June 04, 2022 @08:29AM (#31619542)
Blocking the user agent in my WAF is the only thing that actually worked. So, I'm sure you're right and I am just a random outlying case.
[ Parent | on HN ]