News for hackers. Stuff that matters.
Re:Kubernetes Is Hard by overgard on Monday March 27, 2023 @05:50PM (#35333196)
I agree with the point that production is hard. There's so many things you just don't think about as a developer that end up being important. Log storage, certificate renewal, etc.

I think how "hard" kubernetes is depends on how deep you go. If you're building a cluster from scratch, on your own hardware, setting up the control plane yourself etc. it's very very hard. On the other hand, if you're using a hosted service like EKS and you can hand off the hardware and control plane management to someone else, IMO it's actually very easy to use; I actually find it a lot easier than working with the constellation of services amazon has to offer for instance.

I do think there are parts of it where "best practices" are still being worked out though, like managing YAML files. There's also definitely some rough edges. Like, Helm charts are great... to use. They're an absolute nightmare to write, and there's all sorts of delightful corner cases like not being able to reliably upgrade things that use StatefulSet (last I used anyway). It's not perfect, but honestly if you learn the core concepts and use a hosted service you can get a lot out of it.

Re:Kubernetes Is Hard by ActorNightly on Monday March 27, 2023 @06:15PM (#35333506)
I disagree that production is hard in itself, I think the way people approach production makes it needlessly more difficult. It seems that when launching a service, everyone seems to try to get the entire stack in one shot, which is difficult.

A much better approach is "1. Make it work 2. Make it good 3. Make it fast". Your initial prototypes need to define the core functionality, and then you incrementally build stuff on top of that.

Re:Kubernetes Is Hard by YetAnotherNick on Monday March 27, 2023 @06:39PM (#35333789)
> 1. Make it work 2. Make it good 3. Make it fast

Disagree. The first two objective clashes very much with each other. I have worked in teams that did that. And generally if anything non trivial is done without planning to just "make it work", it would be more often than not, ridden with many design issues which doesn't allow it to be good.

Re:Kubernetes Is Hard by nerdix on Monday March 27, 2023 @07:02PM (#35334033)
I've never seen anything that couldn't be incrementally improved. There are definitely tons of situations where the additional effort to "make it good" is worth incurring upfront. But you can definitely make something work then make it better. And a lot of times, you'll find that your carefully planned out "good" solution ends up not being all that good and needs incremental improvement. It can be hard to get something right on the first try especially if you haven't done it before.
Re:Kubernetes Is Hard by jakupovic on Monday March 27, 2023 @07:08PM (#35334091)
Fully agree here. Not being abe to incrementally improve something that works would be a contradiction as the original thing never worked at all then.
Re:Kubernetes Is Hard by rektide on Monday March 27, 2023 @08:26PM (#35334144)
I see way more making-poor-decisions underbidding-complexity, then spending years hacking on shit cause it's kind of jank.

When folks have license to do good work, I usually see it going well. If they have some informed opinions & group wisdom at their disposal & which they leverage.

Re:Kubernetes Is Hard by YetAnotherNick on Tuesday March 28, 2023 @12:41AM (#35335990)
I have rarely seen bad but barely workable website become good with incremental improvement. After the product reaches some threshold of users, it is hard to change the code structure in a safe testable way if there is no good way to test it in production like environment, and so developers are rightfully afraid to make any changes. I have seen many examples both internally and externally.

- My bank's website which is a decade old and has millions of users, has tons of issues. I am not even talking about UI or look. It randomly hangs, session disconnects etc.

- ERP tool of one of my previous org sucked, even though used by 10s of 1000s of people and having dedicated team in maintaining.

- One team that I worked with has no proper staging environment, and it would have required synced deployment of dozens of services with many teams working with different language and config. And just this simple task couldn't be completed even with push from upper management, just because of complexity.

Re:Kubernetes Is Hard by ActorNightly on Tuesday March 28, 2023 @02:51PM (#35346004)
If you start designing into the future, you are now trying to make it good before making it work.

Making it work means you start with barebones, just to have a proof of concept that runs, is accessible, is secured, even if you have to click things together and manually ssh into EC2s to launch code. Once you have everything working, then the bigger design should be a lot clearer, in terms of what you have to do to make it good.

Re:Kubernetes Is Hard by lcw on Monday March 27, 2023 @06:36PM (#35333745)
> 1. Make it work 2. Make it good 3. Make it fast

Anecdotally, everytime someone tells me Kubernetes is overkill and then follows your approach. In a year they end up building the capabilities that come out of the box with Kubernetes, and of course they aren't as well thought out. Because they were done adhoc and as needed.

Re:Kubernetes Is Hard by dgroshev on Monday March 27, 2023 @08:55PM (#35334361)
Why is that bad? If the business survived for long enough to have that problem, that's a win, not a failure. Being killed or at least hobbled by unnecessary complexity, on the other hand, is a thing in way more businesses than we like to admit.
Re:Kubernetes Is Hard by Spivak on Tuesday March 28, 2023 @12:28AM (#35335919)
Yep, you can grow to be a huge company with just autoscaling groups and some terraform.

Absolutely painfully boring stack. I’m working at a place now with that stack and we haven’t had a single page in over a year now and we do 5 deploys a day ish.

Re:Kubernetes Is Hard by Veuxdo on Monday March 27, 2023 @06:20PM (#35333576)
Security as well. Now step 0 is "Make it secure". When using microservices connected via the internet, you have no choice.
Re:Kubernetes Is Hard by madeofpalk on Monday March 27, 2023 @06:27PM (#35333651)
> When using microservices

1. Make it work

Re:Kubernetes Is Hard by dgroshev on Monday March 27, 2023 @09:07PM (#35334460)
Security is a spectrum.

You can go from out of the box Debian listening on 22/80/443 directly on internet to WAF, intrusion detection, encrypted overlay networks with CA hierarchy and offline route keys, encryption at rest, elaborate IAM, hardware signed commits, full supply chain audits with every dependency vetted and vendored, multi-access-level logs, mandatory spyware on all corporate gear, and so on and so forth. For most businesses it's more of the former than the latter, and there's definitely no such thing as categorically "secure".

Re:Kubernetes Is Hard by yourapostasy on Monday March 27, 2023 @06:51PM (#35333918)
When I see developers use k8s as “programmable infra replacing on-prem functionality as a baseline”, some time after step 3 is, “make it as cheap as on-prem”. Unfortunately, many of the “move fast and break things” tenor that got the services built in k8s in the first place, baked in decisions that will cost as much to re-engineer into an elastic cloud-savvy form that also saves money compared to on-prem.

This is often from organizational or project management mis-coordination than technical choices the developers are fully aware of. Like a system’s owners not prioritizing putting events of interest into a message queue and insisting upon polling for the information on one end. And an offshore operations team saying learning how to support AWS Lambda/Azure Automation/GCP Cloud Functions is an added cost scope to the contract. So the developer is backed into a corner to run cron/SystemD-based polling off a small EC2. Thousands of decisions like this add up fast, and take a major executive committee commitment to root out tech debt and prevent new tech debt from accumulating to unsustainable levels to prevent cloud initiatives from sinking beneath a mire of expensive small decisions in pursuit of new accomplishments. It is a tough balancing act.

Re:Kubernetes Is Hard by sshine on Monday March 27, 2023 @06:01PM (#35333352)
> If you're building a cluster from scratch, on your own hardware, setting up the control plane yourself etc. it's very very hard.

I'd like to make a distinction:

If you do cluster deployment from scratch, Kubernetes actually gets easier.

Because you get good at it. What it also gets is more time-consuming.

If you aim to replicate all of AWS, then unrealistically so.

The art is knowing when to stop. For a lot of people that's even before Kubernetes.

For others, you can run all your apps in a cluster, except persistent storage.

Re:Kubernetes Is Hard by kazen44 on Monday March 27, 2023 @06:11PM (#35333472)
At the end of the day, getting a stable production environment is simply a tradeoff between the amount of complexity you need to make your infrastructure do what you want, and reducing complexity because it removes failure points from the production environment.

K8s is nice and all, but if all you really need can be solved by 2 VM's and a way to announce an anycast address (or use a loadbalancer if that is not an option), why would i add all that complexity?

Re:Kubernetes Is Hard by b33j0r on Monday March 27, 2023 @06:38PM (#35333777)
For reasons of experience, all I ever want from a system is that it’s reproducible.

I had a vanity website running k8s in a managed cloud. I thought I was backed up by my provider and original ansible deployment, which was of course developed iteratively.

I originally did this mostly to do a practice deployment, and get the workflow.

A few years later, it went down and I didn’t notice for a few weeks. It was too unimportant to invest in monitoring, and not worth it to do over. Redeploying gave me lots of confusing errors (note: I also do this stuff for work)

Frankly, I was surprised that the provider doing updates would make my years-stable site fall over. I haven’t tried that one again for funsies, yet. It’s the vendor specificity that was my d’oh!

Re:Kubernetes Is Hard by sshine on Monday March 27, 2023 @06:27PM (#35333648)
I totally agree.

I've gotten away with docker-compose for container orchestration for the last 16 months.