Fascinating, pardon my ignorance but I’m new to this home brew clustering thing. How is the coordination between the nodes handled for a use case like serving a wordpress blog? Or are we talking about n-1 nodes behind a load balancing master node?

Kubernetes would be my guess... Especially k3s, which has a low footprint is suited for rbpis.

Hey mate. The rig looks awesome. How is the performance output compared to a regular desktop with Ryzen-5/7 or Core-i7 processor in it? Can it run Spark? Apologies if my questions are irrelevant. I'm not sure if Spark/Java works well on the ARM platform yet. Although it'd be foolish on my part to expect too much performance from something like this that costs around $600! It'd be an interesting setup to test parallelism and learn how things work.

Imagine clustering 7 desktops from 2006, same performance.

Hahaha. Yeah, but probably an order of magnitude less power consumption though.

I think the Pi4 would be closer to a 2006 desktop (Core 2 Duo). Pi3 is more like a 2002 or so Desktop, perhaps.

Oh, I didnt even realize it wasnt 4. 4 is on par with 2005 Athlon 64 X2, 3 will be ~half that, somewhere around 2003 Athlon 64.

> How is the performance output compared to a regular desktop

Sorry, but often budget (junk) CPUs are only 10% of the performance of current desktop-class processors.

You can use "openssl speed" to test and compare performance.

When I was experimenting with Intel's Atom netbook chip, it was exactly 10% of a Xeon thread. So it could be used for httpd static content, but not java.

> It'd be an interesting setup to test parallelism and learn how things work.

Long-term, they can be used for DNS, PXE, etc. but most people will not get their money back. If you plan to use docker or k8s, verify they have the right CPU options first.

https://www.openssl.org/docs/man1.1.1/man1/openssl-speed.htm...

https://en.wikipedia.org/wiki/Intel_Atom

Are you sure that it wasn't due to the fact that Intel Atom doesn't have AES instruction set[1]? I think openssl uses it a lot so it would explain this huge difference (since looking at the clocks, AVX/SSEs etc. I would expect at most ~50% difference looking at single core of Xeon vs single core of Atom) unless generations of Xeon and Atom were many years apart. I think newer RbPis CPUs have AES instructions so they could be faster than Atoms in applications that utilise it.

[1] https://en.wikipedia.org/wiki/AES_instruction_set

Thanks man.. I know. I've an old Asus laptop/netbook with an Atom processor. Quad-core I might add. The travesty! It's slow as hell. I bought it because it had an 11 inch screen and I wanted something small. I bought it for AUD $300 back in 2015. I haven't used it more than a month. It's just part of my electronics junk collection. Now a days I'm a lot more careful before buying these things. That's why I've been holding on to the idea of buying a bunch of RPi 4s or Jetson Nanos. It's probably a better idea to get a proper GPU for my Ryzen desktop.

As a general concept, an old low-end laptop can sometimes be usefully repurposed as a low end NAS. It just needs a sufficient amount of ram (depends on the NAS software), and the ability to plug in disks via some means (USB, etc).

It wouldn't be fast, but the battery in it (unless completely dead) even provides some power-outage protection. :)

Very nice work. Super geeky. I would love running a Plan9 cluster on this setup. Any thoughts on trying it out?

I have a similar cluster at home that I mostly use as a media center! One change I made over your setup is using an external NAS for storage. I'm also using k3s for making the whole setup completely portable.

When I need to access my movie library within a deployment, I just mount the folder as NFS volume. It's working quite well and performances are still decent! Maybe something to consider for Leviathan?

Hey there. Thanks for the article!

I wanted to ask you if it is worth it implementing something similar in the cloud to learn kubernetes. The hardware is prohibitively expensive on Brazil, so maybe a cluster of t2.micro on AWS running for the total amount of 750 hours could serve the learning purpose for free

I don't know how it is in Brazil but in Europe and the US offices frequently go out of business that have dozens if not hundreds tiny desktop computers in their bankruptcy auctions. They've got i5 cpu's 4-8gb of ram and 128gb SSD's and go for (much) less than a hundred bucks. Stack 3 of them and you have a kubernetes cluster that you could carry in one hand and I think outperforms the 7 raspberry Pi 3's.

Servethehome has a series of articles about the machines, though they're focusing on the slightly higher end ones.

I think it’s lease companies replacing paid off business PCs. They rent boxes to offices on multi year contracts, collect and recycle them when the term is up.

If you have a PC, then just run it on VM(s) on that. That's what AWS is giving you anyway.

Here's an example: https://kubernetes.io/blog/2019/03/28/running-kubernetes-loc...

Thanks for reading! It's probably worth implementing it in the cloud using the free credits if hardware is prohibitively expensive, at least for the purposes of learning.

If you do intend to self-host some services in the longer term, the cost of running it on the cloud will catch up with the hardware cost in no time at all.

Or you could also checkout lowendbox.com (not affiliated with them). They provide really cheap vps servers.

hey nice one will

> An alternative was to use commercial-grade network switches with 16 ports but that was obviously was out of the question (and budget).

Ebay?

Thanks for the tip. I did consider Ebay, but at that point, I figured it was more trouble than it's worth if I bought them used. Ebay doesn't exactly have the best reputation.

Ahhh. I guess it depends on what you're looking for.

Mellanox cards and related cabling is generally still pretty good (in UK and Australia anyway).

A few years ago I obtained a few 24 port HP Procurve switches (3400cl-24G), with 10GbE (CX4 connectors) on the back.

Those worked pretty well, though turned out to be slightly too noisy for use as desktop switches. ;)

Taking a quick look on US Ebay now, something like this (US$39) might have been workable:

https://www.ebay.com/itm/HP-J9145A-ProCurve-2910al-24G-Ether...

I don't know that seller at all, but ex-enterprise HP Procurve gear should generally be pretty safe. From reputable vendors anyway. :)

For Singapore, I'm not sure. Might have needed to get something from the US shipped over. This one seems like it'd be ~US$80 including shipping, which was probably too much:

https://www.ebay.com/itm/HP-ProCurve-Switch-1800-24G-switch-...

To help with the noise try replacing the fans with high quality ones. I threw some nocturnas into an old managed dell switch and it quieted down immensely.

Just be aware that sometimes they change the pins around from the standard pinout, I blew out one of the fans before discovering this. Fan still works, just lost the rpm monitoring :(

Thanks, very much agree.

I took a naive approach to it before having any clue about electronics, just replacing one of the (3) internal fans with a quieter one of the same voltage. Mostly to see if that would work.

While it functioned, the "fan warning" light was then permanently on (with the replacement fan in).

If I still had the gear and were to do it again, I'd investigate properly and figure out a real solution. :)

Server fans are not cheap, they’re smaller and faster. Look at current and CFM rating. They might look insane from DIY market figures.