That works on bare metal you control. If you rent something with VM / hypervisor the mitigations are important to protect from other VMs on the machine.

Mitigation between VMs is only effective when done from the hypervisor. Mitigations within a VM can only counter intra VM data leaks.

I think that's the smart thing to do. Default to safe, let people evaluate their threat model and disable as needed.

I'm curious, what options did you disable?

Add 'mitigations=off' to the kernel command line (assuming you're using linux)

We did it for most of our GCP images

Right, I'm not sure Spectre is worth worrying about outside of big multi-user systems, or a web browser executing Javascript. Otherwise if you're executing untrusted code, you're in big trouble regardless.