our vms we were running our stuff on was costing my team like 20-30k a month, we moved them to azure pay around 500ish

I moved my app to a 486 under my bosses desk and it was cheaper than azure.

Is that 500, or 500k?

Ah yes, the "actual" way to use cloud is to undergo a major re-write of your application such that it is inextricably tied into the proprietary PaaS options of your cloud provider.

Cloud is amazing for highly variable loads. I also see it as a bit of a luxury service for ops types (like me) - I don't have to go to a DC and manage hardware or deal with a DCops crew, so that's nice, but you probably wouldn't buy luxury cars if you were starting a courier business.

What really annoys me is that few if any clouds actually allow "dynamic scaling" of a single instance running operating system where you can hot-add more RAM or more CPU, with or without a restart.

Some can do it, some require you to basically image the entire machine onto a new one, some can't handle it at all.

I’ve never worked at a cloud company, but I know more than I’d really like about some of the stuff under the hood.

First, changing out the CPU type to something that isn’t almost identical out from under a running VM is a mess and may require degrading the system by removing features from the starting CPU. Switching manufacturers at runtime (AMD vs Intel), while sort of possible in theory, is effectively a lost cause.

To add CPUs, you have to deal with the architecture’s nasty hot-add CPU mechanism, and the guest OS needs to be expecting those CPUs starting from when it boots, and that expectation isn’t free. (The latter is Linux’s num_possible_cpus vs num_present_cpus.)

Adding memory involves getting that memory into the kernel’s memory map in the right places. This is more complex than one would like. Removing memory is worse.

And all this happens, in the cloud data center, on essentially normal hardware. If you want to add RAM, the CPUs need to be on a system with more RAM, preferably on the same NUMA node. And vice versa for adding CPUs. If the tenant is paying for local storage, that needs to move, too.

I get why actual "hot add whilst running" isn't a common feature (VMWare can do it but often does it via various tricks, such as the instance is already 'sized' with the number of CPUs/RAM and some are marked inaccessible).

The thing that really pissed me off is when a cloud provider is UNABLE to add more CPU and RAM on reboot. I even understand having to migrate the instance disks to another host that has CPU/RAM available.

Your frustration with VMs and cloud providers is completely understandable. VMs are essentially entire machines, which makes hardware manipulation quite challenging. However, have you considered using containers instead? They are often easier to manage and offer more flexibility in terms of scaling resources up or down as required. Let me know if you are a developer, as I may have some recommendations to share.

Let's face it, fear of vendor lock-in is almost always bullshit.

Everyone's locked into a vendor at some level or another. Want to move off of Oracle? It's work. Want to move off of servers and into VMs? It's work. Do you want to move off of some Java library like Spring? It's work. Even moving from apache to nginx is work (assuming you're not just serving static files).

Your technology choices always lock you in. You can minimize the cost of moving with architecture, but there's only so much you can do.

The key isn't avoiding vendor lock-in, it's understanding what the tradeoffs are for that lock-in and making sure it's a decision instead of a by-product.

It's like writing your stuff in NEW_TECH_LANGUAGE. Sure it might give you better performance, but now you have a bunch of deployment and maintenance headaches, and your people will cost more and be harder to find. And does it even work/scale in production/HA? How do you monitor it? What kind of weird runtime things are you going to get abused by?

I'd say there is a difference between technical path-dependence/tech debt, and a vendor who's actively able to take technical and legal steps to try and compel you to keep using their services. And obviously you should be getting something for the lock-in, otherwise you're setting money on fire for no reason. The fear is what happens once you are well and truly boxed in - somehow IBM still sells a bunch of mainframes, for instance. (Maybe they're truly amazing machines still; I know nothing about that world).

I do spend a fair amount of time and effort trying to keep my company nimble; we try to use open source software and generic cloud primitives so we can move with some ease if need to. As you say, it is all work.

I've never heard of AWS suing someone to keep using their services.

I wouldn't be surprised if Oracle or any outsourcer did that...but that implies a breach of contract, not a transition after a contract is done.

IBM sells mainframes because they're impossible to replace due to the way they're used. Those big printers that print financials statements apparently don't talk to anything else (funny, I was just talking about those the other day). It's about I/O and throughput with those things. You'd think someone would have come up with a replacement, but presumably the market is so small that nobody cares.

Every cloud vendor charges for egress but not ingress, the more data you onboard the more expensive it is to move off