News for hackers. Stuff that matters.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by drcongo on Friday March 10, 2023 @11:10AM (#35096266)
Ahrefs is like my white whale. I hate them. I can't stop myself from battling them. I hate the fact that they ignore status codes, I hate the fact that they ignore crawl rate specs in robots.txt, I hate the fact that they crawl URLs on my sites that have never and will never exist and keep coming back to do it again.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by nazka on Saturday March 11, 2023 @06:37AM (#35107429)
Why not use Semrush?
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by paraboul on Saturday March 11, 2023 @01:11PM (#35110927)
You don't have to "use" ahrefs for them to crawl your website
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by drcongo on Saturday March 11, 2023 @05:17PM (#35113353)
I don't use ahrefs or semrush, and in fact, semrush's crawlers behave even worse.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by phendrenad2 on Sunday March 12, 2023 @12:59AM (#35117192)
Why not just block any requests with Ahrefs user-agent?