Depends on S3's role in your application. If you actually need unlimited, worldwide-replicated object storage with nearly limitless bandwidth, sure, use S3 and pay for it.

Do you just need a place to stash files so your multiple application servers can read them? An NFS export over a private network can very well be all you need, and you can have that file server back itself up to S3/etc at regular intervals.

Well, you could deploy Minio.

I see Kubernetes as an enabler. Grab some commodity hardware - your own or Linode - and have a cloud experience using open-source components.

I just wish Kubernetes wasn’t so darn complicated.

Or use GKE/AKS/EKS and save on the salaries needed to operate clusters and hardware

That's not what they meant. And, S3-compatible object stores already exist. Ceph, for one.

Cachable, scalable apps that you write as part of a web-centric business are ones that use in-application level knowledge to coordinate with the web framework to use caching services at each level of depth of the stack, be it 3rd-party Cloudflare or DIY CDNs (varnish/squid), memcache/NoSQL/database-level.