there's a cost to engineer something that automatically switches or someone going in and manually changing it. so the spot prices has to be higher than ondemand + switching costs. the new pricing models(a couple years old) though have mostly alleviated this

Because you don't want to be pre-empted.

This is no longer the case. Back in 2018 AWS changed the pricing model so now there's no more bidding.

This means "bidding" more won't protect your capacity, AWS will just take it if they need it.

Prices are now relatively flat, and changing slightly over time by a fraction of a cent per day with a cap at the OnDemand price, while previously there used to be lots of from 0.1x to 10x the on demand price and with swings even multiple times within a day.

This shocks me too. When I first had to design a system around spot instances I assumed it caps out at the on-demand price. But in practice that is very much not the case. The spot price can routinely go above on-demand, with all the downsides of spot still being applied to the instance.

It is because many systems are only built to support spot. When running a workload, they might not want to give up availability so they just pay the higher price. To be fair, this is a game that AWS engineers. They essentially want to kick people off the infrastructure to free it up for other uses. So this is a way to signal how badly you don't want to be kicked off. A lot of times after a price goes above on-demand pricing, it will dip down below fairly quickly as workloads all re-adjust. Some companies are willing to play that gamble, that a short period above on-demand is still cheaper in the long run when you average out the invoice at the end of the month.