News for hackers. Stuff that matters.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by bastawhiz on Friday March 10, 2023 @10:41AM (#35095771)
Ahrefs could probably save another few hundred million if they didn't repeatedly visit the same links over and over indefinitely to find the same error codes or a binary file that they surely don't care about. I see them in my logs for my podcast hosting service, hitting the same 404s for months and months. They end up hitting audio files and downloading (or attempting to download) many gigabytes of content each day. They don't do anything with audio! As soon as they get the HTTP headers, they could say "oops, I don't care about this" and disconnect. And in many cases, they got those URLs from the enclosure tags of RSS feeds... I'm not sure what they were expecting!

In the cloud, not in the cloud, frankly it seems to me like their business would be far more efficient if they tuned what their crawler actually crawled.

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by lazyfanatic on Friday March 10, 2023 @11:05AM (#35096173)
No, throw hardware at it! My life for the last 25 years has been trying to optimize code via I/O.

My favorite design pattern is a for loop that tries once per item and completely fails to never try again, nor report it failed at all. /s

Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by drcongo on Friday March 10, 2023 @11:10AM (#35096266)
Ahrefs is like my white whale. I hate them. I can't stop myself from battling them. I hate the fact that they ignore status codes, I hate the fact that they ignore crawl rate specs in robots.txt, I hate the fact that they crawl URLs on my sites that have never and will never exist and keep coming back to do it again.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by nazka on Saturday March 11, 2023 @06:37AM (#35107429)
Why not use Semrush?
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by paraboul on Saturday March 11, 2023 @01:11PM (#35110927)
You don't have to "use" ahrefs for them to crawl your website
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by drcongo on Saturday March 11, 2023 @05:17PM (#35113353)
I don't use ahrefs or semrush, and in fact, semrush's crawlers behave even worse.
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by phendrenad2 on Sunday March 12, 2023 @12:59AM (#35117192)
Why not just block any requests with Ahrefs user-agent?
Re:Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud by hinkley on Friday March 10, 2023 @01:55PM (#35099183)
I cannot express emphatically enough how powerful the concept of classes of service can be for simplifying your production issues.

I think people see the notion of a network neutrality as an all or nothing thing. Network Neutrality between networks, yes, but within your racks you’re trying to balance a large equation instead of two or three smaller ones.

If they reclassified sketchy pages they wouldn’t have nearly as much of the sorts of trouble you mention.