I loved this and watched it to the end.
To any that feel the need to hide or disparage this because it seems to promote doing things on your own vs in the cloud, this isn’t some tech ops Total Money Makeover where you read a book and you’re suddenly in some sort of anti-credit cult. This is hard shit, and it’s the basics of the hard shit that I grew up with as being the only shit.
Yes, you can serve your own data. No one should fault you for doing that if you want. It takes the humble intelligence of the core team and everyone at IA to pull that off at this scale. If you don’t want to do the hard things, you could use the cloud. There are financial reasons also for one or the other, just as there are reasons people live with their family, rent, lease, and buy homes and office space- an imperfect analogy of course.
I hope that some of the others that could go on to work at the big guys or have been working there and want a challenge consider applying to IA when there’s an opening. They’ve done an incredible job, and I look forward to the cool things they accomplish in the future.
I am going to get a transcript done and up soon as well -- I just gave the talk on Friday so haven't had time to do so yet.
[0] http://radar.oreilly.com/2015/04/coming-full-circle-with-big...
[1] https://web.stanford.edu/class/ee204/Publications/Amazon-EE3...
Ideally of course they should be able to meet a fairly wide number of downstream eyeball ISPs at the major IX points in the bay area and offload a lot of traffic with settlement-free peering.
60Gbps outbound from AWS, Azure or GCP would be astronomically incredibly expensive.
It's been a huge blessing for a whole generation of startups to have a radically well-connected space that just about anyone can drop in equipment at multi-gig unmetered (albeit admittedly extremely constrained on power and cooling). It is honestly a part of what has made Silicon Valley great. Well, that and being able to cobble together a few replacement servers from Fry's components (RIP!) or schlep out to some ex-CTO's Sunnyvale garage in the middle of the night to offload some lightly used VA Linux 1U's...
Even today, you can get 15A and a 42U cabinet to call your own with unmetered gigabit for $400/mo - and probably less if you ask nicely.
Because of budget we tend to use things up 100% before putting more in.
And it is not only a cost comparison. You need different kind of people to manage in-house vs cloud, not that you need less or more, just different skills
That's not to say that it wouldn't be incredibly expensive, but probably far less than what you see on the pricing page.
And that right there is why I continue to donate to IA. I am sick and tired of services offloading my data to destructive companies like Amazon.
For example are server logs considered tracking? It seems unreasonable to require that logs not be kept.
Edit: On further thought, I don't even know that tracking should be banned. Instead, I would argue that advertising in the way enabled by tracking should be banned. That way the incentive is removed with less bureaucracy.
We need to kill bureaucratic interests in EVERY THING.
Politicians should be conscripted servants with no method for empowering or financing themselves.
They should run on policy ALONE.
I don't agree there. There is no reason that someone has building bureaucracy as they're goal. The goal is to get something out of it, which is accomplished via the bureaucracy.
> We need to kill bureaucratic interests in EVERY THING.
While I don't necessarily disagree, this is orthogonal to the original issue.
> Politicians should be conscripted servants with no method for empowering or financing themselves.
I don't think conscripted means what you think it means...
Also, that would be impossible to implement. Either you don't try to cut off every path, in which case you have the option of limiting bureaucracy, or you try to cut off every path, increasing bureaucracy.
> They should run on policy ALONE.
I agree. Also orthogonal to the topic at hand. Also, please let me know if you find a way of implementing this without requiring bureaucracy as a critical component.
And yet, the opposite is happening. Government agencies are happy to tap into user data. And sometimes it is mandatory to keep data depending on what you are doing and your country legislation.
AFAIK the US is among the countries where you have the least such requirements, but there are still some sectors where logging is mandatory, like financial services. Many other first world countries (never mind dictatorships) require ISPs to keep data for a year or more.
I guess the biggest downside is the speed of scaling that they can do. As it is limited by how fast they can purchase and install new storage device. But with the use case of Internet Archive, that shouldn't matter much.
Also, bare-metal doesn't necessarily mean owning the hardware. You can rent it too. There are providers that provide bare-metal in one-click and sometimes available within minutes.
It really depends on what scale you're talking about. When you're a startup and suddenly land on the front page of HN, you might need 100x or 1000x your current capacity - in which case AWS will be useful to no end.
If, on the other hand, you're an established name with quite a bit of traffic already and the maximum uptick you will reasonably experience is 2x-3x, the argument holds far less water.
I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather than having its storage and processing hosted by, for example, AWS. His answer: lower cost, greater control, and greater confidence that their users are not being tracked.