I’ve never worked at a cloud company, but I know more than I’d really like about some of the stuff under the hood.

First, changing out the CPU type to something that isn’t almost identical out from under a running VM is a mess and may require degrading the system by removing features from the starting CPU. Switching manufacturers at runtime (AMD vs Intel), while sort of possible in theory, is effectively a lost cause.

To add CPUs, you have to deal with the architecture’s nasty hot-add CPU mechanism, and the guest OS needs to be expecting those CPUs starting from when it boots, and that expectation isn’t free. (The latter is Linux’s num_possible_cpus vs num_present_cpus.)

Adding memory involves getting that memory into the kernel’s memory map in the right places. This is more complex than one would like. Removing memory is worse.

And all this happens, in the cloud data center, on essentially normal hardware. If you want to add RAM, the CPUs need to be on a system with more RAM, preferably on the same NUMA node. And vice versa for adding CPUs. If the tenant is paying for local storage, that needs to move, too.

I get why actual "hot add whilst running" isn't a common feature (VMWare can do it but often does it via various tricks, such as the instance is already 'sized' with the number of CPUs/RAM and some are marked inaccessible).

The thing that really pissed me off is when a cloud provider is UNABLE to add more CPU and RAM on reboot. I even understand having to migrate the instance disks to another host that has CPU/RAM available.

Your frustration with VMs and cloud providers is completely understandable. VMs are essentially entire machines, which makes hardware manipulation quite challenging. However, have you considered using containers instead? They are often easier to manage and offer more flexibility in terms of scaling resources up or down as required. Let me know if you are a developer, as I may have some recommendations to share.