Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by lazyfanatic on Friday March 10, 2023 @11:05AM
(#35096173)
No, throw hardware at it! My life for the last 25 years has been trying to optimize code via I/O.
My favorite design pattern is a for loop that tries once per item and completely fails to never try again, nor report it failed at all. /s
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by drcongo on Friday March 10, 2023 @11:10AM
(#35096266)
Ahrefs is like my white whale. I hate them. I can't stop myself from battling them. I hate the fact that they ignore status codes, I hate the fact that they ignore crawl rate specs in robots.txt, I hate the fact that they crawl URLs on my sites that have never and will never exist and keep coming back to do it again.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by nazka on Saturday March 11, 2023 @06:37AM
(#35107429)
Why not use Semrush?
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by paraboul on Saturday March 11, 2023 @01:11PM
(#35110927)
You don't have to "use" ahrefs for them to crawl your website
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by drcongo on Saturday March 11, 2023 @05:17PM
(#35113353)
I don't use ahrefs or semrush, and in fact, semrush's crawlers behave even worse.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by phendrenad2 on Sunday March 12, 2023 @12:59AM
(#35117192)
Why not just block any requests with Ahrefs user-agent?
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
by hinkley on Friday March 10, 2023 @01:55PM
(#35099183)
I cannot express emphatically enough how powerful the concept of classes of service can be for simplifying your production issues.
I think people see the notion of a network neutrality as an all or nothing thing. Network Neutrality between networks, yes, but within your racks you’re trying to balance a large equation instead of two or three smaller ones.
If they reclassified sketchy pages they wouldn’t have nearly as much of the sorts of trouble you mention.
In the cloud, not in the cloud, frankly it seems to me like their business would be far more efficient if they tuned what their crawler actually crawled.