In those scenarios, the _real_ cost isn't the $5/mo VPS but the managing upgrades, security patches, etc - which would be mostly handled for you in this scenario.
I built a simple email verification service for A company I used to work for. Used API gateway, lambda, s3, and sqs. Few thousand users a month. I don't need to maintain it, check for security updates etc.
The monthly bill for all that was ~$2 NZD.
I run a bunch of stuff on a $5 VPS and more still on small machines like raspberry pi's and so on behind it.
On my distro at least, spinning up a real paravirt container (not docker, a full OS with systemd, SSH, etc) within a VPS is a matter of unzipping a bootstrap to /var/lib/machines and running 'machinectl start whatever-you-called-it'.
It's just a different set of abstractions to learn, and at least IMO one that's more valuable.
Say you have an idea for a web app that needs a server-side component, but you only ever expect to have a few users. A VPS will cost you at least $5 a month. If you can make your backend serverless, you'll probably only pay a tiny sliver of that, if you even end up exceeding your provider's free tier.