News for hackers. Stuff that matters.
Re:RaspberryPi 4 home lab with LXD cluster by herpderperator on Friday October 02, 2020 @11:00PM (#24669129)
What a coincidence, I just finished setting up my 64GB RAM / 2 TB storage Raspberry Pi 4 PoE cluster tonight (only 1 power cable!): https://www.dropbox.com/s/um5mjt8bucfkrbk/picluster.jpg?dl=0

I was really impressed that I got line speed Ethernet between the Pis, 112MB/s doing a wget of a 1G file from one to another. Things have improved a lot since the original Raspberry Pi from 2012.

I need to figure out how to get the 8th Pi working though, since I'm using one of the PoE ports to power a Mikrotik that I'm using as a wireless bridge onto my existing broadcast domain. Technically I don't need the bridge, since each Pi can connect to my wireless AP over 5GHz 802.11ac just fine, but it feels nicer to funnel upstream data through one point. More importantly, it reduces WiFi collisions since there wouldn't be 8 devices broadcasting simultaneously and interfereing with each other (especially at those close distances) if they all needed to download something from the Internet at the same time... like a Docker image for a Kubernetes deployment :)

Re:RaspberryPi 4 home lab with LXD cluster by moray on Saturday October 03, 2020 @06:13AM (#24670953)
Nice project! I just started researching the pieces I need to build something similar. What kind of PoE hat did you use? I was also thinking of having one of the PIs have also an SSD with bootp/dhcp and let the others boot with PXE.

I had been thinking of building a PIs cluster for a while now and I was looking with interest at the turingpi board project [1] they are taking preorders for the second batch if anyone is interested. But I would like nodes with more than 1GB of RAM so I guess I'll have to wait for that.

[1] https://turingpi.com

Re:RaspberryPi 4 home lab with LXD cluster by herpderperator on Saturday October 03, 2020 @06:48AM (#24671109)
This is the PoE hat I used - I specifically chose it because of its low profile and fanless design: https://www.amazon.com/gp/product/B07WD7HXSQ/
Re:RaspberryPi 4 home lab with LXD cluster by hiq on Saturday October 03, 2020 @05:32AM (#24670788)
Have you tried to experiment with 802.11s and mesh networks to remove some of these Ethernet cables by any chance? I'm looking into some "advanced" router features these days, and I was surprised that there is a standard for this, although it seems badly supported. That would seem like a good option for a more flexible (in terms of networking) cluster if the nodes are that close to each other that wireless networking shortcomings aren't much of an issue.

Of course that's only the theory...

Re:RaspberryPi 4 home lab with LXD cluster by herpderperator on Saturday October 03, 2020 @06:55AM (#24671137)
I couldn't tell by your question if you were referring to the Pi cluster or asking an unrelated question about 802.11s mesh clusters... :)

Here are my answers for both:

Pi cluster: meshing won't save any additional wires here since there is already a wireless bridge set up (between the black Mikrotik device pictured, and an off-camera Mikrotik wireless router.) I chose to wire the cluster simply to guarantee a reliable connection within the cluster. You can absolutely use a Raspberry Pi over WiFi without an Ethernet cable, it would just be a less reliable connection (jitter, congestion), and then of course it would have to be powered conventionally rather than via PoE.

General 802.11s meshing: If all you want is fewer network cabling in your home specifically between switches and access points, then it could certainly help. However, the best choice network-wise is almost always a proper deployment with a central router and several APs (broadcasting the same SSID) wired back to it.

Re:RaspberryPi 4 home lab with LXD cluster by hiq on Wednesday October 07, 2020 @08:35AM (#24707539)
I guess I was curious about both, thanks for yours answers! I hadn't thought about PoE, having never used it myself, so indeed at one cable per device there's not much more to do.
Re:RaspberryPi 4 home lab with LXD cluster by jacquesm on Saturday October 03, 2020 @05:19AM (#24670722)
That looks gorgeous.
[dead]
Re:RaspberryPi 4 home lab with LXD cluster by herpderperator on Saturday October 03, 2020 @02:24AM (#24670040)
As others have correctly pointed out, this is a fun, ongoing project on server clusters built using real hardware with real limitations. My laptop from 2018 has 64GB RAM... and would it even matter to say that I already have several high-performance machines? I am not sure what your point is. I did not buy Raspberry Pis to save money, I bought them to have a bunch of small physical servers to play around with. I could have easily started 8 VMs bridged to the network in the same way with virt-manager/QEMU and got 8 much more powerful Linux instances for free if that's what I wanted.
Re:RaspberryPi 4 home lab with LXD cluster by fapjacks on Saturday October 03, 2020 @03:45AM (#24670355)
Like looking at a jeweler's workshop and asking "Why all these shitty little hammers? Don't you know how well a sledge works?"
Re:RaspberryPi 4 home lab with LXD cluster by hddherman on Saturday October 03, 2020 @02:07AM (#24669949)
I'm sure that if their goal was a fast server with a modern CPU, then they would have gone for it.

I see their project as a fun hobby. It's very fun to put a lot of tiny low-power computers together in a cluster and learn as you go. Sure, you should take into account the limitations, but you should not feel discouraged if all you want to do is to have fun, learn and build something that is quite unique.

Re:RaspberryPi 4 home lab with LXD cluster by jacquesm on Saturday October 03, 2020 @05:21AM (#24670733)
And I'm sick of people pissing on other people's nice projects out of envy or malice. Why don't you simply fuck off? Flagged.
Re:RaspberryPi 4 home lab with LXD cluster by oarsinsync on Saturday October 03, 2020 @05:20AM (#24670727)
In terms of raw compute, you're absolutely right.

In terms of understanding distributed computing and the failure modes that can occur, the GP's approach is definitely better. If you build distributed systems only testing them through virtualisation, you'll run into all sorts of issues that come from adding real world constraints that are otherwise masked through the virtualised abstraction.

Re:RaspberryPi 4 home lab with LXD cluster by barumi on Saturday October 03, 2020 @05:27AM (#24670767)
This post just adds noise to an otherwise interesting discussion while missing the whole point of learning how to set up a cluster.

Even the tone itself is non-constructive, all wrong and off-putting.

I'm sure you can understand that someone who bolts on computers over WiFi and praises a 100MB/s connection is not aiming for performance.

This is not what HN is all about.

Re:RaspberryPi 4 home lab with LXD cluster by panpanna on Saturday October 03, 2020 @02:08AM (#24669950)
Most pi clusters are created for learning devops theologies.

Speaking of that, happy to see LXD on the front page. Its not as hyped as Docker but I really like it.

Re:RaspberryPi 4 home lab with LXD cluster by danmur on Saturday October 03, 2020 @02:06AM (#24669937)
Burn :P

One small thing they're nice for is if you want to play with a cluster (e.g. nomad, kubernetes, whatever) on a budget. Of course it's not going to be great bang for buck just for performance but if you're trying out some distributed computing it's nice to be working to real limitations, network and RAM and particular.