Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by drcongo on Friday March 10, 2023 @11:10AM
(#35096266)
Ahrefs is like my white whale. I hate them. I can't stop myself from battling them. I hate the fact that they ignore status codes, I hate the fact that they ignore crawl rate specs in robots.txt, I hate the fact that they crawl URLs on my sites that have never and will never exist and keep coming back to do it again.
[ Parent
| on HN ]
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by nazka on Saturday March 11, 2023 @06:37AM
(#35107429)
Why not use Semrush?
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by paraboul on Saturday March 11, 2023 @01:11PM
(#35110927)
You don't have to "use" ahrefs for them to crawl your website
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by drcongo on Saturday March 11, 2023 @05:17PM
(#35113353)
I don't use ahrefs or semrush, and in fact, semrush's crawlers behave even worse.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by phendrenad2 on Sunday March 12, 2023 @12:59AM
(#35117192)
Why not just block any requests with Ahrefs user-agent?