A big problem is local storage provisioner for kubernetes is still not GA. I want to use an SSD attached to one master, and use that as storage for the other RPIs. Doing this is still undocumented/the Wild West.

Another frustration with the RPi cluster is the arm requirement. If you're only building/running your own stuff, that's fine, but finding containers and helm charts that support arm can be frustratingly difficult.

This is speaking from experience. I love my k3s RPi cluster, it's fantastic once you get things working. I just had to augment with some x86 nodes in order to _also_ run some software that just wouldn't run anywhere else.

Local Persistent Volumes though has been GA since 1.14 and I think entered beta in 1.10 or 1.12, it should be usable now

I was just about to try to start using this, have you seen this article: https://vocon-it.com/2018/12/20/kubernetes-local-persistent-...

The strategy seems to be to create a storage class per app and make sure each persistent volume claim binds a distinct storage class. It sounds like a lot of heavy lifting but it's just a few YAML files... the alternative seems to be https://github.com/rancher/local-path-provisioner which uses the same strategy under the hood to fulfill PVCs as they come online using the Local Persistent Volume strategy, but does not require the storageclass arrangement I understood from the tutorial linked before.

If you want to use an SSD attached to only one special/master node, I don't think that's the wild wild west, I think that's actually closer to what you might see in a traditional network storage architecture, possible to use something like Portworx or Rook to make that work if they are supported on RPi. That's not the same as local storage provisioner though,

If you know more about it than I do I'm happy to hear more about where you got stuck, since I'll be trying to implement this strategy for myself on a non-RPi cluster with a stable set of nodes soon (stable as in, pets not cattle).

Disclaimer: k8s beginner here.

As far as I understand, local storage provisioner is for the node local storage; the storage doesn't follow the workload to whatever node is the pod scheduled, but schedules the pod to the node containing the storage device. It doesn't allow for pods to access local storage outside the node.

So for worker nodes using storage on the master node, isn't it better to use either iSCSI or NFS?

This is exactly right. Local storage is conceptually the persistent volume equivalent to an emptydir volume mount.

If you want to have a 'storage node' in a simple way, the NFS storage provider is the way to go. You install the nfs client libs on each node, setup an NFS share and configure and run the provisioner[1].

My experience with iSCSI is to stay the heck away from it. It is not what you want. iSCSI is really meant for people who already have iSCSI SANs and not people who have a disk they want to share. The more I learned about it, the more I learned that I should have picked something else for every use. It's not that it's bad, it's that it solves a much different problem than I expected given the networked nature of it.

[1] https://github.com/kubernetes-incubator/external-storage/tre... (I think this is the right one, been a while).

NFS-Provisioner and NFS-Client have been around for a while. They're about to be promoted from Incubator actually.

That is fantastic, I remember when they were new ideas without any implementation but just a few people trying some things on GitHub issues.

https://engineering.bitnami.com/articles/scaling-wordpress-i...

This article does the NFS approach justice I think, I was pleased to find it has been a working strategy for a while!

I am not a kubernetes expert so I have no idea. Sounds like a fun challenge. I remember seeing something about network mounts for linux in general. Maybe go around kubernetes? If you solve it I'd be interested in how you did it, so an update would be greatly appreciated.

Kubernetes supports pretty much any type of storage through API Extensions so network mounts are supported. People often use the NFS-Client & NFS-Provisioner projects for this.

Have you used OpenEBS?

Would you not want to stay with an Intel-based CPU?

https://hackernoon.com/diy-kubernetes-cluster-with-x86-stick...

Why?

Because 99% of the time building images for Arm processors is a pain in the ass. It's not just about making sure you have all the requirements lined up, but the processors themselves are pretty slow when compared with a relatively modern Intel CPU, so builds take way longer. Compiling Tensorflow, for example, can take days on a Raspberry Pi.

Source: my company has to build software for Arm on a regular basis.

Only a clam seller would come up with that ;-)

I like it! ^^

It took me a lot longer than I'd like to admit to get what you mean ;)

except it may not easily survive a power outage

Power outages aren't common enough here that this is a big concern. And since the network is likely to be down, there is likely zero point in keeping your server online. (Your services can survive a reboot cycle right?)

If the services are required to be online, it's likely you'd want your network online as well so you'll need a battery backup for your router. So you run the Raspberry Pi on the same battery backup you have your router on.

I believe running a MacBook as an always on server will pretty rapidly destroy the battery anyway.

Macbook appears to be pretty intelligent about it's battery. My 2016 is still over 90% healthy and regularly gets over 5 hours on a single charge (load dependent of course). It's pretty much on 24/7.

The issue is that the battery never gets a chance to discharge if you’re using it as a server. Unless you’re unplugging it and monitoring the battery life and plugging it back in every day.

Hmm, most of my work laptops never get used unplugged and I've never had issues.

It might be fun to implement some sort of service that checks battery life once per day and turns off a smart plug (which the Macbook power adapter is plugged into) and lets the Macbook drain once per day before turning it back on, though.