Knowing what I know now, I think if you are successful something like k8s is required to scale and you are running non trivial apps. For trivial stuff I'd go serverless plus some kind of managed app platform as much as possible.
For non trivial, there are stripped down k8s distributions out there that would work and let you work in portable containers. I'd think pretty hard about something like Rancher to get started, and 'graduate' to GKE / EKS / AKS when you hit scale with at least 100's of VM's. You need to keep it in containers to plausibly make the switch though and not require a rewrite later. I think Rancher deploys on all the major and smaller cloud vendors, is small enough to run on your desktop, still do development, and adheres to the API standards you'll care about later.
Have ansible fire up a docker container on your host.
That's it, done. You don't need Kubernetes.
Using EC2 segregates everything nicely out of the box. You don't need to be concerned about resource contention or figuring out how to size instances/pods. A container going nuts and filling up your disk won't cause a bunch of other services to experience errors.
While convenient for deploys, the biggest issue with Kubernetes is that you introduce an entire new layer of technology that you need expertise in.
If you're running your own EC2 instances anyway (which seems likely, since spot pricing is amazing compared to managed stuff) you now have to know how autoscaling works, plus a VPC, plus Kubernetes. The way Kubernetes interacts with cloud providers (at least AWS) adds a ton of complexity.
Also, if you have a decent build process for AMIs you can insulate yourself from container registry outages by baking the images into an AMI. There's something really powerful about this approach. If you build an AMI properly, that thing will run _forever_ with a minimal set of external service dependencies.
I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
This is borderline trivial. If you build an AMI, you update a launch template with the new AMI and use instance refresh (https://aws.amazon.com/blogs/compute/introducing-instance-re...) to replace instances. Rolling back is the same thing, you just set it back to the previous AMI. Instance refresh took way too long to become a thing, but it was not even very difficult before that. List instances in your autoscaling group, mark them unhealthy one by one.
Now, you'll probably want a couple other safety checks in your deployment process to make sure it all goes smoothly. But you're realistically looking at what, a couple of days of work to have something built to your specifications that can deploy any of your services without introducing the added complexity of Kubernetes?
> I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
The harm is not that the deployments are hard. In fact, they're quite nice. The problem is that small teams don't have the bandwidth to keep up with the relentless release cadence. They may not have the expertise to get the upgrades done confidently.
There is also the fact that, when something does go wrong, you've got a whole lot of complex Kube stuff to churn through.
None of that applies to managed Kube, but then you're paying extra for Kube every month. By contrast, you could have just paid extra one time for a relatively simple deployment script in CI.
About the only real issue I encountered was with major API change involving ingresses, when the cluster would continue running, but we had to update our manifests that were hamfistedly done back in 1.4 days.
What you mean is probably without vanilla k8s
Honestly, it's a lot simpler to either do PAAS, lambdas, or even something like raw EC2 instances to begin with.
The road to containerization and full devops is something that can happen after you've got a working app out in the wild (hopefully) making money.
PAAS tends to lock you in. You don't notice it happening as you add features, then you discover that escaping the platform requires significant re-engineering. (When freelancing, one of my more lucrative gigs was helping a company de-Herokuize their app after the original developers moved on).
Raw EC2 instances usually end up with wizards SSHing in to deploy & tweak. It's sooo tempting.
Having participated in all of these scenarios in the past, I'd recommend any sized company go right to containers. It's the straightest road from zero through MVP to scale, and the best for onboarding new developers.
That said, don't get sidetracked! Pay a company to manage the orchestration. And don't burn time on accessories like CI pipelines or blue-green deploys until the important stuff works.
Same for raw EC2 instances. When you start reaching double-digit numbers, it gets annoyingly harder especially if they aren't identical (for example, due to having different applications, or applications that can't live together, etc)
Getting something up, running, and making money is really the first priority for a business. Fixing it to not suck or eat all the profits comes later.
And we would have close to zero efficiency benefits related to developing the applications, for the price hike that would allow us to hire dedicated engineers to fix issues that would be either common with PaaS or help us making everything work fine on k8s.