News for hackers. Stuff that matters.
Re:Kubernetes reinvented virtual machines in a good sense by jstx1 on Sunday July 31, 2022 @06:03PM (#32299286)
I don't think it's fizzling out, it's kind of becoming a basic standard. And if your use case is simple enough to not need Kubernetes, then doing it with Kubernetes isn't that much more complicated either.
Re:Kubernetes reinvented virtual machines in a good sense by jiggawatts on Sunday July 31, 2022 @06:44PM (#32299578)
A standard in the Linux world. Windows doesn’t have “distributions”, so there’s less need for containers in that space. Microsoft uses Service Fabric internally for the Azure back-end, which covers most of the other use-cases of Kubernetes.
Re:Kubernetes reinvented virtual machines in a good sense by Jenk on Sunday July 31, 2022 @06:53PM (#32299623)
Service Fabric needs to die. It is an abomination on software.

Why, why, *WHY* does each SF service need to have some terrible wrapper code that forces it to run in, and only in, SF? Oh ya.. vendor lock-in.

No thanks.

Yes, I'm aware that MS added support for docker containers in SF. That integration is dreadful. They did about as good a job of it as the WCF team did of shoe-horning "RESTFul Services" in.

> Windows doesn’t have “distributions”, so there’s less need for containers in that space.

I don't follow. What does distributions have to do with need for containers?

Re:Kubernetes reinvented virtual machines in a good sense by jiggawatts on Sunday July 31, 2022 @08:02PM (#32300153)
> What does distributions have to do with need for containers?

A lot (but not everything).

It's important to recognise where the "stable API/ABI boundaries" are. In Linux, the stable boundary is the kernel. Userspace is unstable. There are many distributions, with many small and big differences. Different C runtimes, different folder structures, different configs, modules, etc...

In Windows, the stable boundary is the Win32, COM, and .NET APIs, which are in user space. The kernel boundary is not stable[1] however, which also matters.

Even Linus Torvalds has trouble distributing his hobby dive-computer software to Linux![2] He has no issues with MacOS and Windows, because they have stable APIs and distribution mechanisms. With Linux, the kernel is stable, but essentially nothing else is. If you want to distribute software to everyone, then this is a complex problem with many moving parts that you have to manage yourself.

Containers sidestep this by packaging up the distribution along with the software being deployed. This works because the kernel ABI is stable.

[1] In the Windows world you don't really need to package up the "Windows distribution" because there is only one: Microsoft Windows. Conversely, Windows containers aren't truly portable because the Windows kernel ABI isn't stable. However Microsoft changed this with Windows 11 and Server 2022, the kernel ABI is now "stable", or at least sufficiently stable for Server Core containers to be portable for 5+ years as-is.

[2] He's complaining about "desktop applications", but the exact same rant would apply to server software also. This one talk made me understand containers and why they're so important in the Linux world: https://www.youtube.com/watch?v=Pzl1B7nB9Kc

Re:Kubernetes reinvented virtual machines in a good sense by moondev on Sunday July 31, 2022 @08:55PM (#32300547)
Even if Windows containers are not as portable, the ability to manage them from the same Kubernetes controlplane that manages Linux containers seems pretty powerful. It also enables you to do it outside of Azure/Service Fabric on other clouds or metal.

https://kubernetes.io/docs/concepts/windows/

https://docs.microsoft.com/en-us/azure/aks/windows-faq?tabs=...

Re:Kubernetes reinvented virtual machines in a good sense by jiggawatts on Sunday July 31, 2022 @09:01PM (#32300594)
You're trading one convenience for a world of hurt if you use Kubernetes with Windows. Just take a casual glance at the "Windows containers" issues list on GitHub: https://github.com/microsoft/Windows-Containers/issues

Some of the very recently closed issues were jaw-dropping, such as totally broken networking in common scenarios.

DNS resolution is very different in Windows compared to Linux, making much of the "neatness" of Kubernetes pod-to-pod communication not work.

There is no maximum memory limit in Windows kernel "job objects" (equivalent to cgroups), so one memory leak in one pod can kill an entire node or cluster. This is very hard to solve, and I've seen it take out Service Fabric clusters also.

Etc, etc...

Re:Kubernetes reinvented virtual machines in a good sense by moondev on Sunday July 31, 2022 @09:20PM (#32300728)
Thanks for sharing, it all seems terrifying but still very interesting to explore. I was curious how the Windows nodes would even join - turns out they support cloud-init just like Linux machines.

https://cloudbase.it/cloudbase-init/

In theory this means they could be managed then by something like KubeVirt instead of treating them like a worker node. https://kubevirt.io/user-guide/virtual_machines/startup_scri...

Excited to see this space continue to evolve for sure.

Re:Kubernetes reinvented virtual machines in a good sense by jiggawatts on Sunday July 31, 2022 @11:08PM (#32301507)
Cloudbase-init is interesting, but Microsoft doesn't support or (apparently) use it. There's no mention of it in their docs: https://www.google.com/search?q=site%3Adocs.microsoft.com+%2...

New Windows VMs are generally built using PowerShell DSC or a similar native tool.

Re:Kubernetes reinvented virtual machines in a good sense by mr_toad on Sunday July 31, 2022 @09:05PM (#32300631)
> In Linux, the stable boundary is the kernel.

Different distributions run different kernels. A container from a different distribution will probably work, until it doesn’t.

Re:Kubernetes reinvented virtual machines in a good sense by Maursault on Sunday July 31, 2022 @11:05PM (#32301489)
> Different distributions run different kernels.

Not so much, no. Linux is the kernel, and all Linux distros, every single one, employs Linux as its kernel, or else it would not be Linux. Stepping back from my pedantry, I believe what you must mean is that different Linux distributions will customize the kernel a bit. That doesn't change what it is. Just because you have automatic windows and I have cranks, and you have leather interior and I have plush doesn't mean our vehicles aren't the exact same year, make and model.

Re:Kubernetes reinvented virtual machines in a good sense by mr_toad on Monday August 01, 2022 @12:44AM (#32302167)
They customise, backport patches, and use different kernel versions.

If you run kernel version A in docker, and try and run a container that was designed to use kernel version B you may run into problems.

Re:Kubernetes reinvented virtual machines in a good sense by Maursault on Monday August 01, 2022 @03:12PM (#32310771)
Sure, I can imagine, but that is merely a bug, or perhaps the compatibility issue stems from misuse of a general but inaccurate claim of "Linux compatible." My expectation is specific compatibility is stated plainly in the documentation and on the box, if there is one, without ambiguity.
Re:Kubernetes reinvented virtual machines in a good sense by josephcsible on Monday August 01, 2022 @12:55AM (#32302217)
Note that because of Linus's strict policies on userspace compatibility, A>B is basically guaranteed to always be safe, and only A<B will ever cause problems, unless distros add patches that break things.
Re:Kubernetes reinvented virtual machines in a good sense by AlphaSite on Monday August 01, 2022 @12:59AM (#32302235)
I e heard this before, but the only case I’ve heard of people running into issues is if you hit a kernel bug or depend on a really new feature that’s not present in the kernel yet.
Re:Kubernetes reinvented virtual machines in a good sense by voidfunc on Sunday July 31, 2022 @10:57PM (#32301441)
Kubernetes is heavily used internally at MS as well, especially for newer stuff.
Re:Kubernetes reinvented virtual machines in a good sense by jiggawatts on Sunday July 31, 2022 @10:58PM (#32301449)
Are they running Windows on it? If so, they're very brave people.
Re:Kubernetes reinvented virtual machines in a good sense by voidfunc on Sunday July 31, 2022 @11:25PM (#32301656)
Mixed workloads. Not everything inside MS is Windows, in fact, more than people probably realize is running on Linux. We have our own internal Linux distribution (Mariner) and own Kinvolk (which maintained CoreOS/Container Linux/Flatcar Linux).
Re:Kubernetes reinvented virtual machines in a good sense by x86x87 on Sunday July 31, 2022 @08:31PM (#32300344)
Lol. Famous last words. IMHO K8s is a trojan horse from Google. They are irrelevant in the cloud space and are trying to stay in the game.

K8s is not Borg. Not even close. It's a toy that has limited use.

The revolutionary tech is the Linux kernel features that enabled containers, overlay filesystems and docker image format.

Re:Kubernetes reinvented virtual machines in a good sense by michaellee8 on Sunday July 31, 2022 @09:45PM (#32300910)
That famous tech called cgroup was actually a Google contribution. But I agree that k8s is essentially Google's step to make themselves relavant in Cloud. They have missed the initial opportunity by promoting their PaaS AppEngine instead of something IaaS like ec2 in the beginning of the cloud competition, so Google just play the open-source game and keep releasing stuffs that can be used in all three clouds to lure people to use GCP. But then k8s is a very nice piece of tech that allows one to manage large clusters without vendor lock-in.
Re:Kubernetes reinvented virtual machines in a good sense by x86x87 on Sunday July 31, 2022 @10:50PM (#32301389)
Cgroups, namespaces, apparmor/selinux, overlay filesystems, there is much more to containers than just cgroups.

The no vendor lock is looks great on paper but you are locked in day one. (Eg on aws you probably use IAM, LB, ASG for K8 Nodes - you can maybe move it to another cloud but the effort is going to be significant). Cloud agnosticism is a lie.

Re:Kubernetes reinvented virtual machines in a good sense by vbezhenar on Monday August 01, 2022 @01:56AM (#32302568)
Effort will be significant for any global changes for non-trivial software. Significant effort is fine. Can you compare moving something from AWS to GCP for Kubernetes and for something like Lambda+Fargate?
Re:Kubernetes reinvented virtual machines in a good sense by mhoad on Monday August 01, 2022 @05:01AM (#32303575)
I'd say not only is it a lie but is actively a not very good strategy to pursue right now, at least not all out as though you were pursuing some kind of multi-cloud end game.

At the K8s level, it seems like the introduction of the Gateway API is probably a good level of abstraction to work towards that will keep things about as flexible as possible without all of the insanity that comes with going beyond that to keep everything 100% vendor neutral.

Re:Kubernetes reinvented virtual machines in a good sense by mhoad on Monday August 01, 2022 @04:52AM (#32303531)
It's very strange some of the phrasing of this post. I don't know AWS and Azure that well so I might be off here but every time I look over into those ecosystems it's hard to not come to the conclusion that GCP is literally several years ahead in terms of what they are offering.

Even just sticking strictly to the K8s space, there is nothing even close to matching GKE standalone, let alone GKE autopilot that I am aware of. In the world of serverless the gap looks to be even bigger.

I couldn't imagine a situation where I would even consider something else in a greenfield project / company honestly.

Re:Kubernetes reinvented virtual machines in a good sense by afiori on Tuesday August 02, 2022 @06:58AM (#32317151)
I have seen it described as: "Google thinks that GCP technical superiority is self evident and so pushes a lot less in marketing/developer relations".
Re:Kubernetes reinvented virtual machines in a good sense by rossmohax on Monday August 01, 2022 @05:30AM (#32303726)
> That famous tech called cgroup was actually a Google contribution

At least some of namespaces were contributed by OpenVZ

Re:Kubernetes reinvented virtual machines in a good sense by aledalgrande on Sunday July 31, 2022 @09:32PM (#32300821)
You do know that you can use k8s on AWS, DO, Azure & your own bare metal right?
Re:Kubernetes reinvented virtual machines in a good sense by x86x87 on Sunday July 31, 2022 @10:30PM (#32301256)
Should you use it though? That's the question!
Re:Kubernetes reinvented virtual machines in a good sense by vbezhenar on Monday August 01, 2022 @01:57AM (#32302571)
Why not? Kubernetes brings lot of useful features.
Re:Kubernetes reinvented virtual machines in a good sense by aledalgrande on Monday August 01, 2022 @11:38AM (#32307606)
that is up to you, your knowledge and your needs
Re:Kubernetes reinvented virtual machines in a good sense by tjr225 on Monday August 01, 2022 @12:02AM (#32301916)
Most people using k8s have are not consuming google products on an “enterprise” scale.