Totally agree that this is impressive, but overkill. I've been running a linux NAS with a few services (transmission, sab, nzbget, the other usual suspects) for over a decade now, starting with on-host services and later migrating to docker+compose (which greatly simplified the bootstrapping process). Recently migrated to TrueNAS to get an even more appliance-like setup, and was very impressed in how quickly I was up and running from scratch. I can have NextCloud deployed in about 10 seconds if I want. Maybe I'm just at the point where I don't want to have to tinker/micromanage my server, but the idea of running kubernetes for a home server gives me anxiety.

I run microk8s at home not because it’s easier, but because I want to retain the skills/knowledge if I ever have to work somewhere where k8s is used again. Given how ubiquitous it has become there’s a decent likelihood of that being the case if I go looking for work as a backend software engineer. It also has a couple of benefits in that if I ever decide to move my stuff to a cloud provider like DigitalOcean it’s probably going to be dead simple compared to setting up and reconfiguring something running in a VM.

Yeah, I tried swapping my home server stuff to Kubernetes and regretted it. Not only was it much more complex and indirect, but it required more maintenance. After some months on Kubernetes my home lights, which were driven by a simple daemon, stopped working. Kubernetes was on fire, and it was hours of pasting random incantations to get it back up again. I ripped out Kubernetes, set it up in docker-compose, and it's been fine since.

What was the issue with Kubernetes?

The above question in a way is also my point which more bluntly can be said as, don't run something for the sake of running it and instead run tools you're knowledgeable in.

> Kubernetes was on fire

It might have been something as easy as conflicting IPs on your network that was causing the control plane not get kicked off or the nodes constantly disconnecting.

Something somewhat similar happened to me where my IoT lights caused a thundering hurd kind of problem on my network ip space related to some internet disconnect and I guess a bug in the code. They just kept conflicting IPs with each other and would just loop over the ip space they had. Never dealt with this before and just unplugging the whole switch for 5 mins resolved whatever infinite bug they had.

I’m not sure I follow. Do you mean that you use containers but without Kubernetes? If so, that sounds like a wise choice. AFAICT, Kubernetes is meant for really massive-scale deployments, so it seems quite clear to me that using it for home use is overkill.

Yes, that was my conclusion as well.

A general pattern I see in software is that it scales up much better than it scales down, and Kubernetes is definitely like that. I had hopes it would be useful and educational to run for personal projects, but it's just too heavy.

Interesting observation. I think you’re on to something there.

Kubernetes is actually a set of interfaces. In this case, they are not using default Kubernetes implementation but a simplified one : k3s so the experience could be different. Or not. But it certainly can't be judge on wpietri experience as is.

Honestly, Proxmox + TurnKeyLinux LXC for Docker + Caprover does everything I could ever want. If I need a VM, I'll just spin it up in Proxmox.

Also on Proxmox, but run separate LXCs for each docker container. Lets me do per-“application” QoS/monitoring as Proxmox is also running my firewall.

Am I missing out?

Offtopic question now that you mentioned running a Linux NAS... Which hardware are you using? Because I'm interested in building a home NAS but I don't seem to find any small box ready to hold several disk that isn't ridiculously expensive.

I have had Fractal Design's Node 304 as a home server / NAS for the last 3,5 years. It's quite reasonably priced, the provided fans are quiet enough and can be replaced later on with Noctuas. It doesn't offer disk hotswap but otherwise it's quiet and I can suffer the downtime ;-) + can handle 6 3,5 HDDs :)

https://www.fractal-design.com/products/cases/node/node-304/...

For the past 7 years I run Ubuntu + ZFS with a Xeon E3 + ECC RAM + SuperMicro MoBo inside a NORCO 20-bay 4U chassis. This was a great setup for years since I needed 40TB+ and at the time of build 2TB drives were the best price per gb. Now, I’m moving back from a house to an apartment, so I need something much more compact and quiet and went with a TrueNAS Mini XL, since it’s one of the few tower-like form factors with enough hot-swap bays for me. It runs nice and quiet, and comes with 10GigE. TrueNAS (the OS) has been a pleasure to use too, but it’s obviously a bit more limited than a full Linux install. I now have it loaded with 10TB drives.

ZFS is the hero here. I tried many solutions in the past (mdadm on ext, btrfs, XFS, hardware RAID) and got burned by rotten bits and confusing UX leading to user error data loss. But in the past 7 years running ZFS I haven’t lost a single bit, and been continuously impressed with the incredibly easy to use CLI and “intelligence” of the file system itself. Snapshots, send/receive, datasets, everything is just so well refined. It felt ahead of its time a decade ago, and still does today. Moving the file system from Linux to the FreeBSB based TrueNAS was effortless.

I run FreeNAS (FreeBSD based) on HP Microservers. ECC RAM, 4 drive bays, ...

The older Gen8 boxes Just Work.

With FreeBSD, the newer Gen10 boxes would hang the first time you boot, you need to pause it and set the hw.pci.realloc_bars loader variable to 1. Haven't tried a new install recently. Details here: https://www.virten.net/2017/10/fix-for-freenas-on-hpe-micros...

Not “small”, but I vertically mounted a 4u chassis (front-side down so cool air comes in from the bottom and hot air is exhausted out the top) and it actually feels not much bigger than a standard ATX tower, except I have 9 HD bays.

The case was $120CAD, and the mobo/cpu/ram (i3) were another maybe 300.

Caveat: I’m not running ZFS, and just do classic “data lives in two places on-site and one off-site” backup

I used Helios 4 (32TB) and currently building a Helios 64 (80TB) NAS. It works great with armbian, which was the key for me.

I'm running a Dell T140.

It's not the cheapest, but has idrac "remote control bios". So I never have to sit next to the box. It houses 4 disks + you can add SSDs via PC and it has ECC RAM. Plus I added a couple of years "Next business day on site" warranty, so I'm pretty safe against long downtimes.

I recently bought a Chenbro SR30169 for my backup storage. It’s super compact but also quite noisy.

Instead, I’d recommend getting a µATX tower with sound dampening preinstalled. The extra interior space also makes installation and maintenance a lot easier.

IMO K3S is quite low maintenance and easy to setup. You just need to run a one liner to install and run the cluster. My only gripes is the internal certs only has 1 year expiration and to rotate it you'll have to restart the cluster a few months before the expiration date (it won't rotate the cert unless you restart it near the expiration date). It would be great if the internal cert has long expiration time (RKE has 10 year expiry by default) or provides manual control for rotating internal certs.

until something breaks - then you are busy parsing megabytes of logs and jump from one vague github issue to another - to be fair I've attempted to install in into an lxd container but debugging this complexity monster is nightmare.

If you were just running it without k8s you would still need to generate SSL certificates and setup Nginx (aka ingress) if you wanted secured inbound traffic.

Also CI/CD is completely optional.

I'd only need to get one certificate for Nginx and another for the SMTP server, which are two commands with Lets Encrypt. And setting up nginx as a reverse proxy for an application is as easy as adding two lines to a configuration file. It's far simpler, easier and better documented, and if any problems pop up it will also be easier to debug. You only need to know nginx to solve issues and add things: with the k3s setup it looks like you need to know nginx and Kubernetes.

Or just use caddy and get SSL included within seconds...

> All of that to deploy a bunch of standard applications in a single server, with very low load

> So now, not only do you have to worry about the applications themselves and their configuration, but you also have to worry about all the layers on top of them.

The biggest benefit of K3s here imho is not scaling or performance but to have a standardized API to deploy standardized packages/containers/deployments (Docker/Helm) to. So instead of configuring and maintaining one of the many Linux flavours (and all those layers) out there I have one standard system to worry about.

I have a K3s Rpi cluster running for a few months now. Setup was trivial and maintenance as well. If I have a problematic node I just remove it from the cluster, reflash the SD card with a new install of K3os and put it back in. No state or further configuration to worry about.

All my previous homelab setups where either hand-crafted snowflakes or configuration managed by some tool (Puppet, Salt or Ansible). Each comes with its own problems but they all have the problem that they accumulate state over time and become to hard to manage.

But you have a cluster, which is already different from a single node. There I see how a layer to manage several nodes starts making sense. But when you're managing a single server, you already have a standardized API to deploy standardized packages (APT/YUM + SystemD probably). It's just a different one. Of course when you deviate from that it starts getting messy, but that happens with everything.

I also have a single node 'cluster' for the more bulkier (diskio) stuff that won't fit the raspi's. And it's nice to have the same API accross all my setups. Apart from the data (backups, photos, media, etc) that is stored on the disk there is nothing of state worth saving on the node. If my root disk crashes, I just install K3s again and apply all configurations from the yaml files on my workstation and K3s pull everything back up as it was before.

As you said Linux does offer standardized packages, but they are not applications/deployments. To get them running, beyond installing the binary, still requires a lot of configuration. Nginx (proxy, tls), a database maybe, storage/LVM, firewall, etc. So you quickly run into tools like Puppet and Ansible to manage this. The disadvantage is that they don't reverse the changes they make. So if you want to try something out and deploy it with Ansible, there is no easy trivial way to undo it except for reverting all the changes individually. Also there is always the tempation to quickly tweak something by hand, forgetting to commit it to CM.

With a system like K8s, everything (container, volume, ingress) is a declarative configuration and the 'system' works to converge to the state you declared. If you delete something, K8s will revert all changes. So there will be no lingering state or configuration left. Making everything way more managable imho.

> All of that to deploy a bunch of standard applications in a single server

usage: deploy_app "commit message, push right to master!"

    deploy_app () {
      git add --all :/
      git commit -m "$1"
      git push
    
      ssh user@server <<EOF
        cd app/ &&
        git reset --hard &&
        git clean -df &&
        git fetch &&
        git reset --hard origin/master &&
        sh scripts/deploy.sh
      EOF
    }

deploy.sh

    #!/bin/sh
    set -e
    # java
    sh scripts/rebuild.sh java-api
    # node.js
    sh scripts/rebuild.sh api
    sh scripts/rebuild.sh cron
    sh scripts/rebuild.sh ui
    sh scripts/rebuild.sh windows-api
    sh scripts/rebuild.sh ws
    # terraform
    sh scripts/apply.sh
    # nginx
    docker restart nginx
    # sql
    sh scripts/migrate-database.sh

Rebuilds Docker images, taints the Terraform Docker resources

I have something very, very similar for a new line of business app I've just started building, except with no docker or terraform (I use docker to run a local postgres, that's it).

looks like you're using terraform to provision your docker containers locally? What is the benefit over using docker-compose? Or do you also provision other things?

I’m working on building out the same thing right now (except I’m running it on a RPI cluster in my office), and the objective is my own amusement and enjoyment, (which naturally tends to lead me toward overengineering) but also to learn Kubernetes better.

Reminds me of simpler times not too long ago, when we used to run an entire startup by simply rsyncing files to production via Gitlab CI after tests passed after every commit.

I'm doing that right now, but I can't figure out how to avoid downtime

You make a lot of great points here. Almost all of the features you call out are actually provided by an open source project called Kalm (https://kalm.dev).

Certs, ingress, cicd integration etc. all come out of the box

"by an open source project called..." - claim it. It's your only submission on this portal anyway.

It blows my mind that people set up personal stuff like this when something like "git push heroku master" does the same thing.

For the cost of one Heroku Standard 2X worker, you can get a DigitalOcean managed k8s cluster with 5x 2GB, 1CPU nodes, and that's capable of running basically all of your hobbyist traffic - not just a single service. Heroku is nice, but it's not the same thing.

For hobbyist traffic, you don’t need a standard 2X though.