News for hackers. Stuff that matters.
Re:Internet Archive Infrastructure by benaadams on Tuesday March 02, 2021 @01:48AM (#26312560)
I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...
Re:Internet Archive Infrastructure by walrus01 on Tuesday March 02, 2021 @02:34AM (#26312798)
On the scale of big hosting operations, 60Gbps outbound is not that much. If you're buying full tables IP transit from major carriers at IX points, I've seen 10GbE for $700-900/mo, and 100GbE circuits for under $7k/month. Of course you wouldn't want to have just one transit, but I'm fairly sure if somebody said to me 'here's $20,000 a month to buy transit', on the west coast, it's within the realm of possible.

Ideally of course they should be able to meet a fairly wide number of downstream eyeball ISPs at the major IX points in the bay area and offload a lot of traffic with settlement-free peering.

60Gbps outbound from AWS, Azure or GCP would be astronomically incredibly expensive.

Re:Internet Archive Infrastructure by tutfbhuf on Tuesday March 02, 2021 @06:09AM (#26314036)
It seems that companies can be too big for the cloud and too small for the cloud (don't need k8s). I wonder what's the sweet spot.
Re:Internet Archive Infrastructure by 867-5309 on Tuesday March 02, 2021 @06:53AM (#26314307)
horizontal and vertical scaling are the latest push, but diagonal would be the sweet spot
Re:Internet Archive Infrastructure by jonah-archive on Tuesday March 02, 2021 @10:40PM (#26324447)
Exactly this. There are some logistical complexities (e.g. some of our bandwidth is funded by the E-Rate Universal Service Program for libraries which runs on a July-June fiscal year and so rapid upgrades on that front aren't possible), but by and large egress bandwidth isn't our primary challenge. Intersite links as I noted in the video are the current big one, and that can and does involve occasional time-consuming construction -- but honestly over the past year, a combination of total blowout of my usual capacity planning (including equipment budgets) plus the logistical complexities of lockdown have resulted in slowness to upgrade as fast as we'd like to.
Re:Internet Archive Infrastructure by rsync on Tuesday March 02, 2021 @11:38AM (#26317139)
Two words: hurricane electric.
Re:Internet Archive Infrastructure by dweekly on Tuesday March 02, 2021 @01:02PM (#26318225)
God bless HE Fremont. They are the unsung story of the Internet backbone. If one were to make a list of companies that at some point had a major fraction of their hosted physical infrastructure at HE I suspect it would make people's jaws drop.

It's been a huge blessing for a whole generation of startups to have a radically well-connected space that just about anyone can drop in equipment at multi-gig unmetered (albeit admittedly extremely constrained on power and cooling). It is honestly a part of what has made Silicon Valley great. Well, that and being able to cobble together a few replacement servers from Fry's components (RIP!) or schlep out to some ex-CTO's Sunnyvale garage in the middle of the night to offload some lightly used VA Linux 1U's...

Even today, you can get 15A and a 42U cabinet to call your own with unmetered gigabit for $400/mo - and probably less if you ask nicely.

Re:Internet Archive Infrastructure by owenmarshall on Tuesday March 02, 2021 @08:40AM (#26315169)
IME with cloud in the small and in the large: network prices are artificially high on the cloud providers and are very easy to get discounted if you are a big spender.
Re:Internet Archive Infrastructure by Daho0n on Tuesday March 02, 2021 @10:33AM (#26316310)
So the small pay for the large?
Re:Internet Archive Infrastructure by jeromegv on Tuesday March 02, 2021 @11:09AM (#26316776)
Yes, that's typically how it works for any industry. You get discount on bulk orders.
Re:Internet Archive Infrastructure by xhrpost on Tuesday March 02, 2021 @10:56AM (#26316609)
It seems they are regularly maxing out their network infrastructure. If it's so cheap, how come they don't just buy more? Is it the cost of the actual hardware? (I know they recently upgraded)
Re:Internet Archive Infrastructure by brewsterkahle1 on Tuesday March 02, 2021 @11:24PM (#26324738)
We are upgrading again-- the pandemic has made us kind-of popular.

Because of budget we tend to use things up 100% before putting more in.

Re:Internet Archive Infrastructure by mike_d on Tuesday March 02, 2021 @07:58PM (#26323385)
They are maxing out the fiber links between their own datacenters, which is in the process of being addressed. If the bits can't get from the datacenter full of hard drives to the datacenter that connects to the internet, not much point in buying additional transit capacity.
Re:Internet Archive Infrastructure by charcircuit on Tuesday March 02, 2021 @02:31AM (#26312777)
I have no clue how people can afford what AWS charges for bandwidth. I did the math once for migrating a project to AWS and the bandwidth alone costed 10x my entire current infrastructure for that project, which is something I run for free.
Re:Internet Archive Infrastructure by polote on Tuesday March 02, 2021 @06:55AM (#26314323)
Because people have nothing to compare their AWS cost to. They don't know how much it would cost them to host their service outside of AWS.

And it is not only a cost comparison. You need different kind of people to manage in-house vs cloud, not that you need less or more, just different skills

Re:Internet Archive Infrastructure by blacksmith_tb on Tuesday March 02, 2021 @01:18PM (#26318445)
Absolutely - AWS is this generation's IBM, no one was ever fired for buying services from them, so to speak.
Re:Internet Archive Infrastructure by johannes1234321 on Tuesday March 02, 2021 @08:53AM (#26315311)
Not only that: It is really hard to predict AWS cost. So many variables go in. And starting with a small side project in AWS is easy, and then each additional step is a small step ...
Re:Internet Archive Infrastructure by dilyevsky on Tuesday March 02, 2021 @03:57AM (#26313298)
For lots of orgs I’ve seen it creeps up slowly until you’re paying 10-50x of full transit without any peers but by that point you’re too locked in to do anything
Re:Internet Archive Infrastructure by remus on Tuesday March 02, 2021 @04:07AM (#26313354)
Depends on whether bandwidth is important to what you're doing. In many applications it isn't so even at the inflated prices charged by AWS et al don't really matter in the context of other expenses.
Re:Internet Archive Infrastructure by Sebb767 on Tuesday March 02, 2021 @09:03AM (#26315397)
To be fair here, when you're pouring that much money into AWS you probably have a better contract and can negotiate the price down quite a bit. Additionally, you could use CloudFront to further reduce your bandwidth costs.

That's not to say that it wouldn't be incredibly expensive, but probably far less than what you see on the pricing page.

Re:Internet Archive Infrastructure by tutfbhuf on Tuesday March 02, 2021 @06:14AM (#26314059)
A hybrid solution might be possible. Put your core infra on AWS, but get a very cheap CDN (or custom solution) in front of it to handle the 60Gbps so that only a small fraction will hit your AWS infra. Do the same for storage, e.g. build your own ceph cluster on bare-metal instead of Amazon S3.