Excellent. The biggest obstacle ive had to using Pis for practical things is that the memory cards eventually go bad. Looking forward to seeing this in the main release.

Yes, just yesterday I debugged an issue with my raspberry pi. It would randomly reboot itself about half of the time I ran a certain workload on it.

After swapping out the SD card with a new one, it appears to be fixed. This was hard to diagnose as there was nothing in system logs / error messages about what might have went wrong.

Putting /var/log/ somewhere else mitigates this.

Couple more things to remember:

- don't use swap

- make sure you have noatime on all your filesystems

- identify other folders with frequently changing data (/tmp, $HOME/tmp, etc.) and also make sure you keep them in memory

- you can use NFS for many of these things if that is possible

Something I never understand:

Why is this not the default?

So many people struggle with the same stuff over and over again, yet, there's no mechanism in Linux distros to package default settings for a use case.

You want a stable, easy-to-use platform. Setting /var/log to something other than the main disk risks all kinds of difficult to diagnose problems. If that means burning through SD cards, so be it. It's relatively easy to mod it to write elsewhere, that's exactly the purpose of user customization.

What they ought to do is put an easy way to set that in `raspi-config`

You can solve most of these problems by not using an under powered or bad power supply. From what I have seen most raspberry pi users love to just take a random charger and plug it into their raspberry pi. Micro usb has been known to be totally inadequate as a power delivery mechanism for SBCs and on the armbian forums people are going to complain if you show them an SBC with micro usb. USB C is a lot better but you are still better off with the official raspberry pi power supply.

What is right for Raspberry Pi might not be right for a desktop might not be right for a server.

Typically things are geared for desktop users who have not much idea what they are doing and everybody else is expected to understand and configure to their needs.

Nowadays distributions started preparing separate profiles (workstation, desktop, embedded, etc.) that save a little bit of work.

noatime is a default for most distributions but a person that does install on Raspberry Pi might be configuring filesystem manually and not be aware to put it there.

Swap is very tempting to have on the Pi but is going to kill the card in a hurry.

Eventually sounds like far far away In reality, they really have short lifespan, are slow, and halt the system

Therefore I wouldn’t recommend the rpi anymore.

The whole point of my post is to suggest that it's now easy (soon even easier when it's not a beta feature) to run everything from a durable external SSD. You don't even need a microSD card inserted anymore.

Ideally the next Pi gets some interface that allows more integrated storage, like eMMC or NVMe, but even with USB the speed and durability justifies annoying cabling situations.

the lifespan is mostly due to writes, right? so couldn't you just boot from an SD card but save your actual data on an external drive?

Yes, I have several pis that boot from an sd card and then all of the file systems get mounted over NFS. The sd card only needs to be read from once every few weeks/months when it boots.

Even if it does wear out, storing all the data on NFS (or a usb drive) makes it a lot less worrying.

note this worked on all the previous models of the pi. It just wasn't ready for the pi 4 when it shipped.

Does anyone know if its theoretically possible to run a cluster of Pi's off of the same external ssd at the same time? I'd like for say 4 Pi's to get a speed boost of not running off microSD, but I also don't want to shell out for 4 separate hard drives.

I've looked in the past and have only found links to partitioning the drive so that it could be used when moving from one pi to another. But nothing about using it concurrently.

I'm not even sure what this would look like physically. Like if a hub exists that could pass traffic to a single drive connected on the other side. Or how the drive would internally separate the writes. But im curious if something like this has been done before. Or why it couldn't be done

PXE Boot w/ NFS root - you won't get the bandwidth (about 112MB/s over gigabit Ethernet), but you'll definitely see the response time improvements (IOPS + latency).

This. I intend to do some more testing with my current cluster to see how much improvement I can get with one Pi 4 serving the traffic (vs each Pi running from its microSD card). And ideally seeing if a faster machine with faster storage could do even better.

If you end up doing this you should definitely make a post about it. I'm sure I'm not the only one incredibly curious on the real world performance and tradeoffs

I'm not aware of any solution that exists to connect a single USB device to multiple hosts simultaneously.

You need a 5th PI in front of the SSD providing a network file service.

Networked storage but that won't be as fast as native drives.

I tend to disagree. It will depend on the network and the storage but it is not uncommon for network storage to actually be faster than local flash card.

At home I have couple of older laptops my kids use for various things. I have installed linux on them and configured to boot over wifi and it actually sped up these machines significantly. I would not advise for everybody to do this, though. The setup is complex (there is no built in support in any distro and you are pretty much on your own) and you need excellent wifi coverage (I have no less than 5 5GHz access points to cover the area densely).

One thing flash card will be difficult to beat is random small accesses if the only thing you care about is latency and not throughput.

It's a gigabit network adapter. Ignoring the extra CPU overhead, latency, etc. that alone limits storage to less than SATA 1 speeds. UASP drives on the USB 3 port can push near SATA 3 speeds with lower latency and less CPU overhead.

>"The results really speak for themselves. For sequential operations, using a USB SSD is 3-4x faster than using a microSD card. And for random access, random reads are a bit faster, but writes are about 6x faster!"

Awesome! Do you think this is feasible also for Raspberry Pi 3B?

Yes. The Pi3 could always boot from USB once you enable[1] it. The difference with the Pi4 is that it's initial boot eeprom is now updatable. The first release didn't include the option for USB booting. It has just recently been added [2].

[1] https://www.raspberrypi.org/documentation/hardware/raspberry...

[2] https://github.com/raspberrypi/rpi-eeprom/blob/master/firmwa...

The other major difference is the Pi 4 has USB 3.0 and the network interface is not on the same bus. So, while the 3/3+ also benefit from USB boot, the 4 can benefit quite a bit further!

Earlier Pis could already run off USB drives, it’s just that they released the Pi4 before they had USB boot in the firmware so it didn’t work, but now it does.

Really great article and very helpful, but where does the 8x faster for writes come from? I make it 6.2x.

Oops! My math must've been a bit broken. I'll update that part.

Thanks! I tried to leave a comment on the article itself but it said I was blacklisted (first time visitor so I think something might be going wrong there).

D'oh; I'm still trying to get my spam settings fixed on the site, after a major upgrade a couple weeks ago. Sorry about that.