However, once you hit significant scale, the lessons from the operational experience of all of the major firms have been pretty consistent: it's cheaper to operate your own data centers than it is to outsource them.

From the perspective of a Fortune (checks list) 15 company: AWS saves us a fortune. No facilities costs all over the world with power/real estate/lawyers to handle local laws. No data center engineers across the globe. A solid discount on list. Consistent bills (thanks to judicious RI buys) and servers that are available in minutes - not weeks (have you ever seen enterprise IT ticketing practices?!)

If we had two orders of magnitude fewer employees/servers/locations AWS wouldn't make sense. But at this scale nothing else makes sense.

AWS was likely just a way to overhaul inefficiencies in a legacy IT org. Someone will be able to do the same thing in 5 years moving you from AWS back to self-hosted.

https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...

But some companies get so big, it actually makes sense to build their own network with their own custom tech and, yes, abandon the cloud. Amazon and Google and Microsoft can keep cloud prices low, thanks to economies of scale. But they aren't selling their services at cost. "Nobody is running a cloud business as a charity," says Dropbox vice president of engineering and ex-Facebooker Aditya Agarwal. "There is some margin somewhere." If you're big enough, you can save tremendous amounts of money by cutting out the cloud and all the other fat. Dropbox says it's now that big.

I think hybrid totally makes sense where you take the most expensive part bare metal with limited scope of maintenance.

1 PB is $23k per month on S3. It's nothing. That's barely the costs of 1-2 employees in SV.

The migration itself would take a lot more effort than one dude, even if there was a solution for completely free storage out there, the migration could only result in a huge net loss.

Bandwidth is almost always cheaper at a colo. Most compute instances are cheaper to buy and rack if you have continuous loads. Disk ... is tricky.

It is faster and has lower up front costs to get clouds up and running initially. For most startups who are going to fail, that's Good Enough(tm).

If, however, you continue existing for a while, the other things start to add up at much lower levels that you would expect. I'd say the crossover is around when you are spending about $15,000 per year. Your colo is about $10,000 of that per year and you can rack 5 new machines every year for the remaining $5K. That's not that much for an actual business.

Cloud is good for your initial startup and for bursty situations. Once you have continuous loads, you need to be moving to pulling stuff back to your own hardware.

It's a real myth that the cloud magically saves me a sysadmin. I have found almost exactly the opposite. Using the cloud effectively takes more time and more expertise. Debugging the cloud effectively takes WAY more expertise. Combine this with the fact that someone has to be able to architect a system to fit within the constraints of being on the cloud, and you're down extra employees.

The difference is in: "Eh, it's been down for 3 days? Sigh. Just reboot it." vs. "Um, that request hung for 93 seconds. Why?"

In the first case, the cloud is fine.

In the second case, someone is going to have to traipse over an enormous amount of systems (which you don't own and can't always instrument) and variables (some of which you aren't even aware of existing) to hunt it down. If, however, you can say "Pull that off the cloud into our own systems and keep an eye on it." you have made your debugging life a lot easier.

Of course, once you have the ability to do that, your team realizes and starts asking: "Given how much time we spend debugging issues with the cloud that aren't actually our fault, why are we on the cloud again?"

I always smile when that realization kicks in. Now I generally have to stop the team from pulling everything off the cloud. However, that's a much easier task.

For the hardware side of things every colo I've seen has a "remote hands" service.

AWS is also about 2749 times faster and more efficient, as measured by me the last time we ordered hardware on AWS and dell simultaneously.