I couldn't help but rewrite the entire thing, and now it's a single 8K SLOC server in App Engine, down from about 70K SLOC.
Someone convinced the right person to “put stuff on kubernetes” and then booked it for another job/greener pastures with barely any production services, or networking configured.
Thus an opening for a new SRE and once again I find myself looking at a five headed monster of confusing ingress rules, unnamed/unlabeled pods, unclaimed volume specs…
Sigh.
Also, there's no inherent lock-in, you can basically just deploy it somewhere else.
Data is where the lock-in lies. Moving can be hard if you use proprietary databases. Can still be worth it.
That means very little, I hope you realize. Reader, Voice, Chat, etc.[0] were all around a long time.
>Also, there's no inherent lock-in
GAE has plenty of proprietary APIs you can depend on. Whether or not you do is up to the programmer.
0 - A comment below notes that voice and chat aren't deprecated yet. Voice was announced deprecated, and google has had so many chat apps I'm not sure which ones are gone. Anyway, here is a more complete list of things Google has abandoned: https://killedbygoogle.com/
There's other reasons to use K8s than just thinking of massive scale.
Setting up environments becomes a massive PITA when working directly with VMs. The end result is either custom scripts, which is a messier version of terraform, which ends up being messier than just writing a couple of manifest files for a managed k8s.
> anyone with K8s experience is demanding $150k + 10% Superannuation here in Australia... minimum!
sheds a tear for CAD salaries and poor career decisions
The author advocates using a high-level PaaS. For sure working directly with VMs is the wrong answer to premature optimization, but as an early stage startup, there's plenty of services around that you basically just need to point your Git repo at and you'll have a reasonable infrastructure set up for you.
I guess I've just never found this to be true.
My main goal when engineering a solution is always, "Keep It Super Simple (KISS), so that a junior engineer can maintain and evolve it."
Working directly with operating systems, VMs, networking, etc. (purely in Cloud - never on-prem... come on it's 2022!) is the simplest form of engineering and is much easier than most claim.
Genuine question, why is this bad? Is it because k8s can spin it up but becomes unreliable later? I think the industry wants something like k8s - define a deployment in a file and have that work across cloud providers and even on premise. Why can't we have that? It's just machines on a network after all. Maybe k8s itself is just buggy and unreliable but I'm hopeful that something like it becomes ubiquitous eventually.
It's everything that comes afterwards that gets neglected. Not by everyone, granted, but by most.
If you need a service to manage K8s for you, then that's a red flag already (regarding K8s, not you personally.) If the service is so complicated that experienced engineers tell me constantly that only managed K8s is the way to do it, that tells me enough about why it's going to be a rough journey that should probably be avoided with IaaS or PaaS.
> ... and wildly a better pattern than just running VMs.
I've never had an issue running VMs, personally. And when I join a firm and begin helping them, I find I can very quickly and easily come up to speed with their infrastructure when it's based on straight IaaS/SaasS/PaaS. If it's K8s, it's often way more complicated ("Hey! We should template our YAML configs and then use Ansible to produce the Helm charts and then run those against the infra!" - ha ha!)
It's really not, k8s does a ton for you, trying to do the same with VMs would be unbelievably complex
> that should probably be avoided with IaaS or PaaS.
PaaS is great till it isn't, seen plenty of companies hit the edges of PaaS and then need to move to k8s
> infrastructure when it's based on straight IaaS/SaasS/PaaS
Again this is great till it isn't (see heroku) and then people move to k8s. Having control and understanding of the underlying infrastructure is important unless your just running some basic web app
Level 2) you pack your software into an ami and use the update the asg config. You can periodically "drain" the asg of old instances
Level 3) you deploy your stack again with the new stack having the ami that you've build at level 2 referenced. You start shifting traffic between the old stack and the new stack. You monitor and rollback if something is wrong.
Now you've got automatic, constantly updating VMs every night if you want them. And a new deployment is just commiting code to master and pushing and that whole pipeline triggers for you.
People like to overcomplicate things, Mirceal. You're on the right path :-)
But to answer the question, VMs have been a thing on the desktop for a very long time.
we also live in a world that does not appreciate well crafted software and a lot of things are driven by the desire to build a resume. I've maintained code that was decades old and was amazing to work with and was still generating ridiculous amounts of money. I've also worked on code that was just written and used all the possible bells and whistles and development speed grinded to a halt once the it's been around for more than a couple of months.
My worst case scenario is having to work on code where the original developer didn't understand what they were doing and they just wanted to use X. Double the trouble if they didn't master X when the thing was put together.
I would very likely agree in all cases.
However, they only represent 4-5 users out of all 3,400. And that's the issue - only a small fraction of the industry operates at that scale :-)
Good luck to them, but they're not going to occupy time and space in my mind.
We'd like that (I'd like that), but resume-driven choices are a very large driver of technology direction, unfortunately.
It means those of who want to build something very maintainable and very stable using the most boring (stable, secure) technology possible are often outnumbered.
We seem to mostly manage to avoid that in my department, but we have a very low turnover.
The code example is not always right.
Beware, if you know it will be needed, you might as well make it a function now. Likewise if you think probably it will be needed, why not make it a function now?
It’s not a good review comment or rejection to say “yeah but I don’t want to do that because it’s not yet needed”. Sure, but what if you are just being lazy and you don’t appreciate what it should look like long term?
The “I don’t want to write a function yet not needed” is not a clear cut example.
I wasn't aware that some devs have a side hustle as fortune tellers?
On a more serious note, you should take a look at Sandi Metz's "the wrong abstraction". https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction
A simple example. If you are making a multi-page website that contains a header on all pages, you can separate the header into a component from the get go, instead of duplicating the header for each page and then abstracting it in the future (where it starts to become more work).
I agree entirely too.
>> Start with a monolith on two VMs and a load balancer. Chips and networks are cheaper than labour,
Kudos to you! You are a dangerous man for you opine the truth.
My advice is generally, "Build something. Then see if you can sell it." or "Sell something and then go build it." Either way, it all starts soooo small that the infrastructure is hardly a problem.
If you "get lucky" and things really take off. Sure. Yeah. Then get a DevOpps superstar to build what you need. In reality, your business will very probably fail.
No. DevOps is about the development and operations disciplines working together in a cross functional way to deliver software faster and more reliably.
In a small enough startup both disciplines may be represented by a single person, though.
News to me. I work at one the biggest companies around, and our DevOps posture is pretty great, and that was achieved by cross functional teams from the historic dev and historic ops teams doing all the things that Accelerate codified.
You don't go from zero to needing global 24x7 support overnight.
Hiring 1 DevOps superstar is exactly what we did a few startups back and it worked great. Of course there was no after-hours support, it's a small startup. Eventually the team grew.
I’d just caution that when it comes to the mental health of the one person in this role, even if they seem like they are doing ok, check in frequently.
>> You can’t just hire 1 DevOps superstar
Secondly, that assertion is not necessarily true. Obviously, you should just hire 1 DevOps superstar... in some cases.
Don't nitpick and don't argue foolishly.
I like to call what the author is referring to as, "What-If Engineering". It's the science of thinking you'll be Google next week, so you build for that level of scale today. It involves picking extremely complicated, expensive (both in compute and skilled labour) technologies to deploy a Rails app that has two features. And it all boils down to, "But what if..." pre-optimising.
It happens at all levels.
At the individual unit level: "I'll make these four lines of code a function in case I need to call it more than once later on - you know, what if that's needed?"
It also happens at the database level: "What if we need to change the schema later on? Do we really want to be restricted to a hard schema in MySQL? Let's use MongoDB".
What's even worse, is Helm and the likes make it possible to spin up these kinds of solutions in a heart beat. And, as witnessed and evidenced by several comments below, developers think that's that... all done. It's a perfect solution. It won't fail because K8s will manage it. Oh boy.
Start with a monolith on two VMs and a load balancer. Chips and networks are cheaper than labour, and right now, anyone with K8s experience is demanding $150k + 10% Superannuation here in Australia... minimum!
https://martinfowler.com/bliki/MonolithFirst.html