A much better approach is "1. Make it work 2. Make it good 3. Make it fast". Your initial prototypes need to define the core functionality, and then you incrementally build stuff on top of that.
Disagree. The first two objective clashes very much with each other. I have worked in teams that did that. And generally if anything non trivial is done without planning to just "make it work", it would be more often than not, ridden with many design issues which doesn't allow it to be good.
When folks have license to do good work, I usually see it going well. If they have some informed opinions & group wisdom at their disposal & which they leverage.
- My bank's website which is a decade old and has millions of users, has tons of issues. I am not even talking about UI or look. It randomly hangs, session disconnects etc.
- ERP tool of one of my previous org sucked, even though used by 10s of 1000s of people and having dedicated team in maintaining.
- One team that I worked with has no proper staging environment, and it would have required synced deployment of dozens of services with many teams working with different language and config. And just this simple task couldn't be completed even with push from upper management, just because of complexity.
Making it work means you start with barebones, just to have a proof of concept that runs, is accessible, is secured, even if you have to click things together and manually ssh into EC2s to launch code. Once you have everything working, then the bigger design should be a lot clearer, in terms of what you have to do to make it good.
Anecdotally, everytime someone tells me Kubernetes is overkill and then follows your approach. In a year they end up building the capabilities that come out of the box with Kubernetes, and of course they aren't as well thought out. Because they were done adhoc and as needed.
Absolutely painfully boring stack. I’m working at a place now with that stack and we haven’t had a single page in over a year now and we do 5 deploys a day ish.
1. Make it work
You can go from out of the box Debian listening on 22/80/443 directly on internet to WAF, intrusion detection, encrypted overlay networks with CA hierarchy and offline route keys, encryption at rest, elaborate IAM, hardware signed commits, full supply chain audits with every dependency vetted and vendored, multi-access-level logs, mandatory spyware on all corporate gear, and so on and so forth. For most businesses it's more of the former than the latter, and there's definitely no such thing as categorically "secure".
This is often from organizational or project management mis-coordination than technical choices the developers are fully aware of. Like a system’s owners not prioritizing putting events of interest into a message queue and insisting upon polling for the information on one end. And an offshore operations team saying learning how to support AWS Lambda/Azure Automation/GCP Cloud Functions is an added cost scope to the contract. So the developer is backed into a corner to run cron/SystemD-based polling off a small EC2. Thousands of decisions like this add up fast, and take a major executive committee commitment to root out tech debt and prevent new tech debt from accumulating to unsustainable levels to prevent cloud initiatives from sinking beneath a mire of expensive small decisions in pursuit of new accomplishments. It is a tough balancing act.
I'd like to make a distinction:
If you do cluster deployment from scratch, Kubernetes actually gets easier.
Because you get good at it. What it also gets is more time-consuming.
If you aim to replicate all of AWS, then unrealistically so.
The art is knowing when to stop. For a lot of people that's even before Kubernetes.
For others, you can run all your apps in a cluster, except persistent storage.
K8s is nice and all, but if all you really need can be solved by 2 VM's and a way to announce an anycast address (or use a loadbalancer if that is not an option), why would i add all that complexity?
I had a vanity website running k8s in a managed cloud. I thought I was backed up by my provider and original ansible deployment, which was of course developed iteratively.
I originally did this mostly to do a practice deployment, and get the workflow.
A few years later, it went down and I didn’t notice for a few weeks. It was too unimportant to invest in monitoring, and not worth it to do over. Redeploying gave me lots of confusing errors (note: I also do this stuff for work)
Frankly, I was surprised that the provider doing updates would make my years-stable site fall over. I haven’t tried that one again for funsies, yet. It’s the vendor specificity that was my d’oh!
I think how "hard" kubernetes is depends on how deep you go. If you're building a cluster from scratch, on your own hardware, setting up the control plane yourself etc. it's very very hard. On the other hand, if you're using a hosted service like EKS and you can hand off the hardware and control plane management to someone else, IMO it's actually very easy to use; I actually find it a lot easier than working with the constellation of services amazon has to offer for instance.
I do think there are parts of it where "best practices" are still being worked out though, like managing YAML files. There's also definitely some rough edges. Like, Helm charts are great... to use. They're an absolute nightmare to write, and there's all sorts of delightful corner cases like not being able to reliably upgrade things that use StatefulSet (last I used anyway). It's not perfect, but honestly if you learn the core concepts and use a hosted service you can get a lot out of it.