You don’t need to reimplement everything aws provides. AWS provides services on demand for a huge costumer base. The sysadmin/devops team you set up needs to solve only your particular problem. You can often get a better, easier to use and maintain system this way. The downside is that you have to pay for that extra team, but if only 20% of your data already costs millions, your scale is big enough that you’ll likely save money by hiring a sys admin team.

a lot of people underestimate how expensive it is just to have a 10g/25g/100g network and maintain it. that alone is extremly expensive. especially when you want it over multiple locations. if you want to connect two datacenters with a low latency high troughput network you would probably go to aws since that is cheaper. and that is just the network, you also need to maintain other stuff like storage. maintaining a storage network is extremly hard. like s3/block storage, etc.

Sometimes, sometimes not. A lot of common ‘prod bricks’ (S3, Managed kubernetes, etc) get used because they are convenient, and while they could be implemented in some other, more bespoke way, it’s rarer and rarer that it actually pencils out as a net win. You also have to deal with the complexity of managing your own version of it, which is non trivial over the full lifecycle of something.

If it is your core business to provide that thing? Sometimes or even often worth it. Otherwise, often not.

Unless you are literally Amazon or big Co, spending ten of millions ( if 20% is millions then at-least 5-10 million ) on just SSDs alone ! should make it pretty much a core business problem to solve?

My sense that in the last decade startups have not lost the skills to do Co-Location setups as they did in 2000s and think it is more complex than actually is. Co-Lo hardware management is hard yes, but if it not even worth doing 10+ million /year budgets we would never have SaaS companies pre-cloud at all.

Which is the core business problem exactly that you think they should be solving?

A core business problem, the core business problem may or may not be the same.[1] At this scale infrastructure cost is a critical problem that can afford to have dedicated teams/vertical trying to solve for.

Hypothetically if you are spending $50 Million+ / Year on the cloud a dedicated team of even 10 senior engineers to setup your co-location with your hardware to consider migration of your costliest and also least cloud native components would maybe cost $2-5M more. With attractive debt financing that is readily available these days you can easily amortize your purchase expenses over the 2-3 year hardware lifecycle and realize savings, there is not much justification not to also pursue this along with all other features you are also pursuing.

The cost is a very low investment compared to your costs with potential for very high saving ROI, so even if the chance of success is low you should give it a shot. i.e. If you can save say $10 Million on the $50 Million, your $2 Million investment needs only 20% probability of success to have expected value in the green.

[1] I don't think there should be only one core (the) business problem for a startup, there are always few critical problems startups have to solve for at any given stage.

The parent was mentioning operational agility, and ability to quickly nearly ‘hot swap’ in live fully formed instances - and with a footprint of multiple petabytes of storage.

Their core business problem is providing databases, and apparently they see leveraging the huge VM and storage pools available at AWS as a major advantage here (and I for one can’t blame them), over hardware spend absolute efficiency.

Being able to providing a couple hundred TB of extra SSD with a config file change (or return it and stop paying for it almost immediately), has real advantages over rolling it yourself, especially if you only have a 10 person ops team or the like.

Considering the apparent business model, I can see their point.

This project being discussed on the thread is likely a couple folks for a few months - low hanging fruit to save millions. What you’re referring to is a major business effort, if not doubling of headcount, for such a company with at best similar payoff. Running their own colos also means a lot of thinking, planning, and lifecycle management when it comes to equipment generations, upgrades, making sure you’ve got the right amount of spare capacity but not too much, etc.

Also, let’s not forget geo/availability zones.

Not saying co-located hardware is not always worth it - rather they seem to be aware of the trade offs, and are making a rational decision based on their business model.

Later, if they have switched from ‘rapid growth and adjustment’ to a more stable state where they can predict things more in advance, maybe they’ll switch. Maybe they won’t.

Like a large energy consumer, running on utility grid at a certain size in a certain area is often much better than rolling your own generation capacity. Sometimes it’s impossible or less cost effective. Sometimes it doesn’t make sense to even try to do the math, and just get hooked up to the grid.

It’s impossible to know the best approach without understanding the specifics. It’s true that adding more resources quickly when you own the hardware will take more, but given how much cheaper it is, you can seriously over provision. Using aws is probably going to be more efficient usage of hardware, that’s why it’s not even more expensive. The thing is that it’s often not the more efficient use of money, specially at that scale.

All these things have tradeoffs, but even with a kubernetes setup there is a variety ways you can configure it that will work better or worse for your workload.

Regarding being the core business, in this regard, that doesn't matter that much. You'll either pay amazon or hire your own team. In either case you'll be spending money in something that's not your "core" product. If you can replace amazon with a bespoke system for a fraction of the price and same resilience, why not?

Someone have said in this thread before, managing those things is not really rocket science. You can have a small, focused team who's able to manage a lot of resources and that can and often is cheaper than outsource. Obviously, it depends in a number of factors. Whether or not it's your core business doesn't seem a decisive one.

Most people (period) will be unable to effectively hire/build/retain a team who can competently build those associated pieces, let alone all of them necessary to operate effectively at this scale. It is why tech is hard, and remains hard, for the majority of companies, governments, etc.

If something is a core part of the business, 1) it’s something they either already have a demonstrated level of competency in, or they wouldn’t be in business, and 2) efficiencies and improvements here should make them more money in a direct and measurable way, and 3) attempting to outsource it exposes them significantly to counterparty risk that can put them out of business, which is generally not considered a good thing.

In some ways it’s like a factory that uses a lot of power. Should they build their own power plant or use the utility. That depends on many factors. Using the utility is often the better choice and works out better, but not always.

If the quality and price of the power is a core part of what makes the company competitive, probably - and that is going to be a key factor in where the factory is located, when it operates, etc.

Where did I gloss over them? I literally suggest spending 2 million a year on staff.

Without specific numbers it is a bit difficult to be as clear as I would like but I read your comment as suggesting that the savings from owning/hosting your own equipment could pay for the team needed to operate that solution -- but then what was the point of switching?

The devil is in the details, and I wouldn't say that it never makes sense to bring operations in-house, but your post didn't make a clear case from my point of view.

Well that's a great point of discussion. My illustration was that you could spend all the same budget and actually have 10 staff and a bunch of real hardware on your balance sheet, instead of just an enormous AWS bill. That's staff that might bring real talent and innovation to your company. And real hardware that will serve you even through financial hard times.

Or you could spend half the budget and in my opinion still be way ahead, but that depends on your execution and the talent pool that's available to you of course.