cwiggs

This was a great read. I've been a sysadmin/SRE/Devops/Platform/infra ( Why are there so many different titles) engineers for many years, for the past ~5 years I've been using Kubernetes and have helped companies migrate lots of workloads to k8s. I've worked with many people who dislike k8s, but almost all of them had almost no experience with k8s. A lot of the anti-k8s people would eventually build out something similar to k8s, which made me love [this blog post](https://www.macchaffee.com/blog/2024/you-have-built-a-kubern...) that helped explain that more.

Anyway, I love that you are/were skeptical of Kubernetes, but then looked into k8s more; we need more people like that in the world.

I think you did a great job explaining the internals of Kubernetes and then comparing it to "not" k8s. e.g. when you said the following:

> Compared to "classic" clusters, a Service provides a similar abstraction to the combination of:

    ** N nodes running Keepalived + NGINX (with a dynamic set of backends)
    ** CNAME records for the virtual (Keepalived) IP

It's a great point because a lot of anti-k8s people just say "Just put your service on a few VMs and put a LB in front of it". But if you don't just pay someone else to do it (ALB,NLB,etc) you gotta do something like Keepalived + NGINX like you mentioned, and then you are one step into creating your own k8s.

I also love how you still brought up issues with k8s. For as much love and experience I have of k8s, I feel like there is some odd behavior it has as defaults. e.g. the default is no cpu or mem requests (don't get me started on storage). I've worked many places where an engineers app that "shouldn't" use a ton of resources used up enough to cause the k8s worker node to have issues, which then caused other pods to fail.

Yes. Even before this you could use MrChromebox's UEFI Coreboot on a lot of ChromeOS devices. You can run Linux or even Windows on them.

If you want something with more power look into the Dell Latitude 5400 Chromebook Enterprise or the Dell latitude 5300 2-in-1 Chromebook Enterprise. Both of these devices have DDR4 user replaceable memory.

There is also the Framework 13 Chromebook edition, but that seems almost impossible to find now-a-days.

Based on Canonical's track record with previous project I'm just waiting for them to give up on Snap.

Hah! Glad I'm not the only one who has this theory https://news.ycombinator.com/item?id=50104114

It's possible this type of "radical" behavior is why this project exists in the first place. I'm willing to bet if the Libreboot maintainer(s) were not "radical" they'd just be using a Mac and be content with a closed source bootloader.

Not to say this "radical" behavior is helpful, and I understand why people don't like it.

Wow, this is awesome! This relates to work I've been doing with the [Heads](https://osresearch.net/) project. Specifically I got a PR merged for the first ChromeOS Port using MrChromebox's code as an upstream dependency: https://github.com/linuxboot/heads/pull/2133

I'm now working on the 2nd ChromeOS device in this PR right now: https://github.com/linuxboot/heads/pull/2225 And there is a 3rd port by someone else: https://github.com/linuxboot/heads/pull/2219

I'm surprised Libreboot was able to add all the ChromeOS devices at once, Just adding the 2 ports for Heads hasn't been easy, although I'm not an expert when it comes to Coreboot.

Anyway, I love seeing more support for Chromebooks, they are great devices. Funny enough my Acer Chromebook (Kano) is now more open and more secure than my Framework 13s.

Curious how you run things and have good work life balance for your employees, with so few people?

Especially with the migration to k8s. K8s is much more complex than Heroku, some even say it requires an entire infra engineering team to manage.

Anyone know how this compares with stackgres and cloudnative-pg?

Looks like this supports mysql and mongodb, which the others don't.

Sure you can use an AWS ASG, but I assume you also tie that into an AWS AlB/NLB. Then you use ACM for certs and now you are locked in to AWS times 3.

Instead you can do those 3 and more in k8s and it would be the same manifests regardless which k8s cluster you deploy to, EKS, AKS, GKE, on prem, etc.

Plus you don't get service discovery across VMs, you don't get a CSI so good luck if your app is stateful. How do you handle secrets, configs? How do you deploy everything, Ansible, Chef? The list goes on and on.

If your app is simple sure, I haven't seen simple app in years.

I think what they should have said is "there isn't a tool like Helm for ECS" If you want to deploy a full prometheus, grafana, alertmanager, etc stack on ECS, good luck with that, no one has written the task definition for you to consume and override values.

With k8s you can easily deploy a helm chart that will deploy lots of things that all work together fairly easily.

It doesn't have to be that way though. You can use the AWS ingress controller, or you can use ingress-nginx. You can use external secrets operator and tie it into AWS Secrets manager, or you can tie it into 1pass, or Hashicorp Vault.

Just like picking EKS you have to be aware of the pros and cons of picking the cloud provider tool or not. Luckily the CNCF is doing a lot for reducing vender lock in and I think it will only continue.

I agree with this somewhat. The other day I was driving home and I saw a sprinkler head and broke on the side of the road and was spraying water everywhere. It made me think, why aren't sprinkler systems designed with HA in mind? Why aren't there dual water lines with dual sprinkler heads everywhere with an electronic component that detects a break in a line and automatically switches to the backup water line? It's because the downside of having the water spray everywhere, the grass become unhealthy or die is less than how much it would cost to deploy it HA.

In the software/tech industry it's common place to just accept that your app can't be down for any amount of time no matter what. No one checked to see how much more it would cost (engineering time & infra costs) to deploy the app so it would be HA, so no one checked to see if it would be worth it.

I blame this logic on the low interest rates for a decade. I could be wrong.

k8s is complex, if you don't need the following you probably shouldn't use it:

* Service discovery

* Auto bin packing

* Load Balancing

* Automated rollouts and rollbacks

* Horizonal scaling

* Probably more I forgot about

You also have secret and config management built in. If you use k8s you also have the added benefit of making it easier to move your workloads between clouds and bare metal. As long as you have a k8s cluster you can mostly move your app there.

Problem is most companies I've worked at in the past 10 years needed multiple of the features above, and they decided to roll their own solution with Ansible/Chef, Terraform, ASGs, Packer, custom scripts, custom apps, etc. The solutions have always been worse than what k8s provides, and it's a bespoke tool that you can't hire for.

For what k8s provides, it isn't complex, and it's all documented very well, AND it's extensible so you can build your own apps on top of it.

I think there are more SWE on HN than Infra/Platform/Devops/buzzword engineers. As a result there are a lot of people who don't have a lot of experience managing infra and think that spinning up their docker container on a VM is the same as putting an app in k8s. That's my opinion on why k8s gets so much hate on HN.

Helm Charts and Terraform are different things IMO. Terraform is better used to deploying cloud resources (s3 bucket, EKS cluster, EKS workers, RDS, etc). Sure you can manage your k8s workloads with Terraform, but I wouldn't recommend it. Terraform having state when you already have your start in k8s makes working with Terraform + k8s a pain. Helm is purpose built for k8s, Terraform is not.

I'm not a fan of Helm either though, templat-ed yaml sucks, you still have the "indent 4" insanity too. Kustomize is nice when things are simple, but once your app is complex Kustomize is worse than Helm IMO. Try to deploy an app that has a ING, with a TLS cert and external-DNS with Kustomize for multiple environments; you have to patch the resources 3 times instead of just have 1 variable you and use in 3 places.

Helm is popular, Terraform is popular so they both are talked a lot, but IMO there is a tool that is yet to become popular that will replace both of these tools.

Most people on Nostr are pro Bitcoin, not pro cryptocurrency. Bitcoin and crypto are not the same. There have been tons of scams in crypto, but not in Bitcoin. Sure there were people who created a scam and told people to send them their Bitcoin (Blockfi, Celsius, Netflix doc "Bitconned"), but that isn't the same as the crypto scams. The Bitcoin core devs never created a token and then ran a pump and dump. It's like saying someone scammed someone with USD, therefore USD is a scam.

This has been my experience as well.

e.g. if you read anything on your local cities subreddit you will think everyone in the city is left leaning. You will think that anyone who isn't left leaning is the devil; they are pedophiles, conspiracy theorists, etc.

If you go out into the city 99.9% of the people are perfectly pleasant regardless of their political leanings.

Very cool stuff. This type of life is why I originally got into tech. Few thoughts:

- I wish this guy had a personal blog that I could subscribe to. I like that post, but I could care less about the rest of the blog posts on his employers site.

- I'd be interested in why he has stuck with a Model Y, seems to not be a good fit. I'd love to see him switch to something better (probably an ICE even though I know tech people love EVs).

All in all kudos to this guy for all the work he has put in to make this life work, love the iterative approach.

All the cloud providers offer a storage class for k8s. The storage class allows you to tell k8s that you want a persistent volume (PV) and it will make API calls to the cloud provider to get you a block storage device. You can tell k8s you want to use that PV in your pod and k8s will automatically mount the block storage to the worker node that your pod lives on and makes it available to the pod.

OP uses Longhorn which is a whole other thing that I've only read about.

For at home you can use other storage classes like ceph, NFS, etc.

IME yes, yes it is. I usually tell people running a k8s cluster is similar to running your own "cloud". If you want to just deploy an EC2 instance you just tell AWS you want an EC2 instance and you are done (mostly). You don't have to worry about if the hardware under the EC2 VM has enough resources, you do have to worry about that with k8s though. If you want to lock down the EC2 VM to have certain permissions you use AWS IAM, with k8s you have to use cluster roles and cluster role bindings. You can apply these 2 examples to many other things in k8s vs "cloud" provider, Ingress, persistent volumes, etc.

> ...the problem arises immediately when you try to resize a data volume and you can't because the API doesn't support it

What API doesn't support it? k8s has support for resizing PVs and has for a while now. AFAIK all 3 cloud providers (and more) support increasing the PV using their storage class.

> K8s is not really for stateful systems, yet

Is this written somewhere or is it just your opinion?

This seems to be the common advise given, but I don't fully agree. There have been many times in my career where a DB was on a VM with the storage attached via a cloud providers block storage. When asked if we should move it to k8s, people are quick to mention k8s doesn't do well with persistent storage. However all of the big cloud providers offer the ability to easily create a persistent volume in k8s that then just creates a block device, attaches it to the k8s host and makes it available to the pod.

So in both situations you have the same IO limits of block storage. The question is does k8s persistent volume api add enough of an IO bottleneck to cause issues, IME that isn't the case.

Now if you want direct attached NVMe drives for higher IO than a network attached block storage will give, then it might be easier with a VM vs k8s, but I can't speak to that much.

Proxmox already has clusters that allows you to live migrate VMs between nodes.

How do you like this setup?

I've been thinking of moving from docker-compose to podman, specifically using the [podman-play-kube](https://docs.podman.io/en/v4.2/markdown/podman-play-kube.1.h...) but haven't gotten around to it.

I like Podman has a lot to offer for self-hosters but it isn't popular (yet?)

I'm sad there isn't any talk about nostr on here. Even if Reddit dies and everyone moves to Lemmy or some other alternative it's only a matter of time before something similar happens to Lemmy. Sure with Lemmy it wouldn't be all controlled by one company, but your data is still locked behind whatever Lemmy instance you posted it on.

Maybe they can find a local Hackerspace that will help get a better dev environment setup?

If the have tech skills I would think they could get a basic job easily.

Generally curious, how are trans people loosing access to health care in red states?

How many servers do you have on this setup? How do you handle persistent storage with docker swarm?

What phone do you have that has a battery that big?

Thing that kills the dells is soldered on ram.

Framework is much better than any of these dells and usually cheaper.