What an entitled bunch
So many downloads.
I'm downright shocked that this wasn't Rick Astley.
Or so he says, it may have been mostly copycats after a time.
Though if shall own any blame it's for being naive and inexperienced in the days before we all needed rate limiting against corporate bots.
As it goes, the whole shebang got archived on Wayback/Internet Archive so all the goodies remain up there for people to enjoy and I stopped needing to run a box and deal with misuse. God bless the Internet Archive I guess.
It's fairly surprising to see that approach from pretty large brands/companies. Some Indian ISPs are kinda notorious in that they'll just link to huge image files on third-party sites and just let their 300 million customers hammer the poor site into the ground. I guess it saves them bandwidth, but they also run a huge risk of pissing someone off and have that asset replaced by something nasty. When people use this kind of hard-linking/hot-linking of resource I don't think they do it to save money or to be evil, I think it mostly gross incompetence.
We had an issue where our product images would just be ruthlessly scraped, if there had been some rate-limiting in place we'd most likely just have allowed it or not noticed. Normally I just pointed the poor scraper to a multi-gigabyte file or funny and unrelated image, if you want to download the same image of a beaver 25.000 times go ahead.
But I also seen amateurishly search engines just pound a site to the point where it was easier to just deny all traffic coming from their crawler, as compared to try to reach out and figure something out. Say what you will about Google and Bing, their crawlers are well written and well behaved.
But Netflix was running at full HD with no problem. So they must have had local cache's with the ISPs.
source: I used to work at Akamai (it was a while ago). The myriad things they do with their edge servers is pretty amazing, but for simple CDN stuff, most of the time clients aren't shipping hardware to isps. That being said, if Netflix can convince an isp to pay its electricity and hardware maintenance costs for a popular isp's customers that's going to pay for itself very quickly. So, i can see why Netflix offers this. It's just not normal AFAIK.
And it's much harder now to have a caching server because everything is HTTPS, so you have to coordinate with both ends to make it work.
Given that Eternal September remains a thing to this day, I wonder if that remains the case?
As for Google Bot being well behaved - I'm glad to hear that this has changed because at the turn of the century (which is the last time I dipped my toe in to self hosting) it certainly wasn't well behaved in the slightest and it would ruthlessly crawl away, happily ignoring any robots.txt limitations applied.
Yes, I know it's also given us CDNs and Single Sign-On, but there are ways to implement SSO as a more active action without that, and I'm not convinced that CDNs are worth the cost we paid.
Eventually the bandwidth was getting hammered by a huge number of leechers seemingly from some apps that had simply hard-linked to the resources.
After replacing said resources [0] they soon ceased but not without a slew of abusive and entitled emails demanding I restore the SFX.
Oh fun times!
[0] https://fukpig.bandcamp.com/track/all-of-you-are-cunts-and-i...