And now, on cloud it’s the same but much more expensive and worse performance. We’ve been struggling for over a month to get a single (1) non-beefy non-GPU VM allocated on Azure, since they’ve been having insane capacity issues, to the extent that even “provisioned” capacity cannot be fulfilled ;-(

Sure, but that’s because Azure. I’m sorry someone made the decision to go there. AWS & GCP, stock outs at least used to be nearly unheard of.

Until you hit a certain scale.

I totally agree about Azure being the worst of the three, they wanted us to commit to certain use before even buying hardware themselves. Crazy…

But I also had capacity issues with Google at large scales in many zones.

What sort of scale, if you don’t mind me asking?

Hey, sure! That’s important context.

One gameserver was 40vCPU and 256G of RAM, we had about 30-50 before we’d see some issues in some regions. (this is from memory now unfortunately).

Sao Paulo and Tokyo being the worst, but Singapore, Australia and Mumbai also had issues at various times.

The other places where we hit hard limits was Los Angeles, but we had more than a hundred instances then.

The issue with the hard limits is that it’ll be one zone thats exceeded and the API will fail, so you have to retry with another zone in the same region, but you don’t get to practice building your autoscaler before you actually need it.

Oh interesting. Yeah, and I’m guessing you don’t get much prior visibility into available stock, since that would also expose info to their competitors.

That is a lot lower than I expected, but I also imagine that’s a sizable order that they like getting.

If you're a top X customer running in a smaller region of AWS or GCP, yes, you need to do capacity planning with your TAM. You get to a point where quota increase requests are not auto-approved.

I think this is partially that for the past decade or two, on-prem was partially preferred by very frugal companies.

One of the places I worked that was on-prem enforced a "standard hardware profile" where the servers were all nearly the same except things that could be changed in house (like RAM sticks). When they ordered hardware, they'd order like 5% or 10% more than we thought we'd need to keep an "emergency pool".

If you ended up needing more hardware than you thought and could justify why you needed it right now, they'd dip into that pool to give you more hardware on a pretty rapid schedule.

It cost slightly more, but was invaluable. Need double the hardware for a week to do a migration? No problem. Product more popular than you thought? No problem.

>I think this is partially that for the past decade or two, on-prem was partially preferred by very frugal companies.

Sure this is made worse by frugality, but I experienced this problem when virtualization was in its infancy, much less cloud anything even existing much less being popular.

There's "cloud" as "server in the cloud I can use" which is what the majority of smaller players are using - it's just someone else's server.

There's also "cloud" as the API-driven world of managed services that drain your wallet faster than you can blink.

This. Instant availability of compute and storage, purchasable with a corporate credit card, was the cloud killer app.

I can't have instant ressources, azure Switzerland is overbooked and can't create much ressources, at two different clients. And I read something similar for AWS.

And in most "large" companies, you still need to go to different teams and processes to have those cloud ressources.

Oh and if your company made the mistake to outsource their IT, odds that you'll have a 4 digit bills to change a teraform.

Welcome to Western Europe.

Seems like a few negotiation skills would be of better use than doing extreme amounts of work so somebody can take months on approving new hardware. Well guess what, the people that did slow down hardware procurement are now slowing down deployment of cloud resources as well because the fundamental problem wasn't addressed and that's organizational misalignment and disfunction.

Even if you can get instant approval, Dell or HP won't Fedex you 50 servers next day like they will 50 laptops.

And once you're a customer you get to deal with sales channels full of quotes and useless meetings and specialists. You can't just order from the website.

You can get servers elsewhere as well and they can be more agile than Dell/ HPE/ SuperMicro/ Lenovo. Asus, Asrock, Gigabyte and some other brands can ship servers rather quickly. Perhaps not the next day but the prices are much lower out of the door especially if you are a smaller customer. Most companies know well in advance how much compute/ storage they will need. So if something takes a month or two it isn't a big issue.

You're going to negotiate with the structure of an entire corporation?

Excuse me CEO your budgeting process is inconvenient to my work, please change it all to be more responsive.

This is not how things work and not how changes are made. Unless you get into the C-suite or at least become a direct report of someone who is nobody cares and you're told to just deal with it. You can't negotiate because you're four levels of management away from being invited to the table.

An organization that can make agile capital expenditures in response to the needs of an individual contributor is either barely out of the founder's garage or a magical fairyland unicorn.

You would be surprised. I have unclogged processes inside and between companies before where people would tell me it's not possible or that success is unlikely.

And yes, if you are working with professionals most expenditures can be planned well in advance so you definitely can take months or even more than a year in the process. If there is a major issue affecting the business of the company you would be surprised what is possible. I've got things approved in weeks in an enterprise of 2000+ employees before and I was 5 levels down from the CEO then. I have improved my negotiation skills greatly since then (it helps when you co-found a consulting company).

> On the cloud it takes five seconds to get a new machine I can ssh into and I don't have to ask anyone for the budget.

This isn't exactly realistic, not for too long anyway.

Once your cloud bill climbs into the millions, expect to see just as much scrutiny on what's costing so much and what can be cut and can you really justify the new thing you want to spin up (as there should be).

Having been through many growing startups, I'd say the freewheeling days of spin up whatever you want only last to about 50K to 100K/month AWS billings.