It sounds like the number of instances needed fluctuates heavily, maybe max number of concurrent instances needed in a month is ~500 64GB instances while average could be ~100 instances?

Spot instances actually make this manageably cheap!

Spot instances for dev environment???? "Your instance will be terminated in 1m" sounds like the new "Can't work; code is building"… haha

We don’t use spot because they can be yanked away easily. Savings plans and hibernation are the biggest cost savers.

How is buying a your dev a single top of the line expensive 64GB RAM MacBook equivalent to on-demand EC2 instances?

Despite containers working locally, the ease of having a remote image with everything preinstalled is huge. Lots of large companies offer internal dev cloud environments - I’ve always found it to be a frictionless experience versus local setup at other companies.

Suppose you’re building a feature that requires $newlib as a dependency. How is that handled? Do you request the addition of $newlib and wait for a new remote image to be deployed for everyone before you can work on your feature branch?

IME these remote instances come with sudo rights, and one is free to install/configure as they please. The initial image can be chosen too.

Right. The devs have full sudo access. They can install and configure whatever they want. Our Bento tool has pre/post setup hooks for all services. So teams can automate this. When something is commonly used we just preinstall it for them. Or some just contribute to the repo. It is all fairly straight forward Ansible and shell scripts. We push nightly to everyone's existing machines.

We actually moved away from containers for dev purposes for almost all of our apps, though many teams still use them. Mainly because they are a bad developer experience. We invested in making apps run consistently in a native way.

Bento was actually our first attempt at doing all of this on our local MacBooks. Many users found that their machine would slow to a crawl and heat up. M1 Macs and now OrbStack instead of Docker Desktop help reduce that. Moving off of docker helped as well. Ec2 gives us the consistent image that is easily testable and repeatable. Egress costs here are negligible.