What’s the difference between a cap limiting your traffic and server cpu maxed out, or db connection pool or ... limiting your traffic?

One bankrupts you, the other just has a minor hiccup with a few lost customers that day.

a (cost) cap is a feature designed to prevent you from going bankrupt but at the expense of not recovering from the surge after you deplete the cap until the end of the cap granularity.

So, if the cost cap is defined as $n per day, once you deplete it you'll be down for the rest of the day (or if you take some manual action to increase the cap, if the cloud provider supports it).

This problem is a function of the granularity. Imagine a system that let you say:

"I want to spend max $n per second with a extra burst of $m per day/week"

You adjust your "$n" to match the throughput you'd have if you had a fixed size "pay what you provision" system and reserve $m for lucky events as landing on HN.

The amount of planning you have to do is similar to the traditional resource allocation, but with the benefit of paying less than provisioned if you're not getting all that traffic.

Is it possible to make a similar approach in cloud? Something like "if traffic goes up, scale up until $5/month and then don't scale further, let it be slow (or even better throw errors)". Would be the best of both worlds.

For services that are pay per request, this is effectively a hard cap.

For more capacity based services, sure, but it's more likely to be down than just slow. When systems are run at their limit they rarely operate the way they did with a little less traffic.

Well, a rogue line could potentially as much shut down your app hosted in DO.

That's the thing though: If you're a big company, your site being down is much scarier than a 100x bill.

When you're an individual, the potential for a $10k bill is much scarier than your hobby project going down.

When you're a small org/startup, the potential for a $75k bill is probably still scarier than your site going down.

Well, then the issue with the Amazon Lambda is not that of it's ability to scale, but that of it's default not to limit your spending.