Surprisingly, the article doesn't seem to mention SquashFS[1] or EROFS[2].
Both SquashFS and EROFS are filesystem specifically designed for this kind of embedded, read-only use case. The former is optimized for high data density and compression, and already well established. The later is comparatively new and optimized for high read speed.[3] SquashFS as a rootfs can already be found in many embedded applications using flash storage and is typically also combined with tmpfs and persistent storage mount points or overlay mounts.
For both those filesystems, one would build a rootfs image offline. In the Debian ecosystem, there already exists a tool that can bootstrap a Debian image into SquashFS[4].
This is actually why I switched all my Pis to Alpine Linux since they use a RAM disk by default for RaspPis Haven't lost an SD card since, meanwhile my Octoprint RasPi has failed 4 times (mostly because of random power disconnects)
I'm a fan of Alpine too, but I want to point out that to the best of my knowledge at least, running root from RAM is trivial and is not something you need to change distros for, you simply have to add the "toram" kernel boot parameter. For example, if your entire read-only root is contained in "filesystem.squashfs", then adding "toram=filesystem.squashfs" would read all of this into RAM on boot.
I can't vouch for every single distribution out there, but I've done this successfully on Debian at least.
This problem seems very overblown to me. The simple answer is "don't skimp."
- Use a power supply that can deliver enough current
- Use good USB cables that can actually carry said current (a lot of people miss this one)
- Do not use cheap SD cards
- Oversize your card so that the card can properly perform wear-levelling.
I have a pair of Raspberry Pi's (2's and/or 3's IIRC) running constantly since 2015. They're running an ancient version of Redsleeve (and are on my list to rebuild but never mind that). DNS, DHCP, and a few other services run on them. They've also survived multiple abrupt powerfail events.
They log to the SD card, and yet I've never had a problem. I personally believe that the fact that I'm using oversized cards with gigabytes of free space is the key. Well, perhaps that and the fact that my personal home directory is on NFS.
While they didn't use tried and proven method that's in raspi-config, it's not about skimping, it's a fundamental underlying change. Your expensive SD will eventually fail. It just will take longer than the cheap one. The question is, when will that be? If you use a read only technique, you don't have to guess.
True enough, but if something breaks with a read-only SD you are relying on logging to a cloud service.
So you reduce the chance of SD failure but introduce a risk of network failure. Of course, network failure doesn't have to be fatal but you won't have any logs to know what happened.
In my experience the chance of one of my Pi's not being able to access the network for one reason or another is much larger than a failed SD card, so I keep my logs on the card.
The last few times I used a read-only rootfs it was to build an appliance-like device which you plug in, do stuff and when its not needed can be unplugged. For this, there's no other solution.
As someone who kept my Pi 1 in a drawer for years and then bought a Pi 4 after they became available again but after building out a bit of a home lab, the internet consensus seems to be you're lucky.
The Pis are energy efficient and worth running apps where catastrophic data loss aren't a concern. I use my Pi 1 with DietPi to run AdGuard and my Pi 4 for simple web apps, like dashboards.
But the world has moved forward and there are options like the ODroid that are almost as cheap and energy efficient, without the relying on an SD card. I run my Home Assistant off my ODroid with nvme. They are all backed up to a couple different places, but my failure expectations are completely different. I fully expect my Pis to just have completely ruined cards one day, whereas I am more worried about a bad update to my Home Assistant server.
> Oversize your card so that the card can properly perform wear-levelling
I think this is half of the problem: SDHC does not require wear-leveling at all, I am unsure of SDXC. But regardless, the wear-levelling algorithm the SD card firmware does is completely invisible, from some vendors it might do a decent job, from some it might be a no-op, we have no way to tell.
The only way to know you're getting wear-levelling is to do it at an OS level, e.g. UBIFS stacked on MTD emulation - but there's a bunch of assumptions there which make it unsuitable for SD cards.
I too have been worrying about SD card failures in my raspberries but haven’t had a single failure ever! My cards are read/write all Debian and have been running since 2016. I hope this post won’t jinx it but my entire data centre has been running with zero storage failures despite running in the most humid and dusty basement you can imagine, though I did have other hw failures such as power supplies, rams, and ups.
Similar. I've been running multiple Pi's for years, and had issues with two.
Got a USB cable tester and found both had cables with 1+ Ohms resistance, which would cause drop to less than 4V under load!
Swapped both cables to some good ones and never had an issue since.
I always use A1 or ideally A2 rated cards from reputable brands bought from reputable stores. Not sure if they help with longevity but the additional IOPS are very noticeable.
I've been running a public facing website on my RPi 2 since 2014 and I am still on the original SD card. But, all assets and logs are being written to tmpfs, and the only thing on the SD card (outside of the OS) is the DB.
Not sure how things are these days, but it used to be the case that power fluctuations could kill an rpi's sd card. So yeah, you could mitigate with a great power supply and a good surge protector, but in practice I found it hard to figure out which power supplies to actually trust.
For my own deployments, I ended up doing read only boot from SD card, making use of a ram disk for temporary files, and hosting the actual os on a USB stick, which was much better shielded from power issues. Worked great, never had issues, but the setup was pretty non standard.
Likewise I have 6 Rpi doing various things all with minimum of 64gb sd cards and official Rpi power supply. Oldest running continuously since ~2014. Never had an issue. I just spend £18 on a 128gb sandisk card from a trusted seller instead of ~£4 on an small cheap one.
There's one (not always but often enough) viable alternative to read-only root filesystems that is often overlooked. It's the possibility to copy the entire rootfs onto a ramdisk when booting. It increases the boot time and obviously disables persisting data (logs etc.), but often these are not needed anyway.
It's just a few lines of code in initramfs!
--- a/usr/share/initramfs-tools/scripts/local 2021-11-05 12:50:23.541088057 +0100
+++ b/usr/share/initramfs-tools/scripts/local 2021-11-05 13:02:14.483203576 +0100
@@ -180,9 +180,20 @@
# Mount root
# shellcheck disable=SC2086
- if ! mount ${roflag} ${FSTYPE:+-t "${FSTYPE}"} ${ROOTFLAGS} "${ROOT}" "${rootmnt?}"; then
- panic "Failed to mount ${ROOT} as root file system."
- fi
+ #if ! mount ${roflag} ${FSTYPE:+-t "${FSTYPE}"} ${ROOTFLAGS} "${ROOT}" "${rootmnt?}"; then
+ # panic "Failed to mount ${ROOT} as root file system."
+ #fi
+
+ mkdir --parents /tmp/diskroot
+ mount -t ${FSTYPE} ${roflag} ${ROOTFLAGS} ${ROOT} /tmp/diskroot
+
+ mount -t tmpfs -o size=6G none ${rootmnt?}
+ chmod 755 ${rootmnt}
+
+ cp --force --archive --verbose /tmp/diskroot/* ${rootmnt}
+
+ umount /tmp/diskroot
+ rm -r --force /tmp/diskroot
}
local_mount_fs()
Even better, you can just put the whole root fs into the initramfs image, and never pivot to another root filesystem during boot. This also allows you to netboot a whole OS.
One pitfall of this is that the decompressed contents of your initramfs must fit within half of your physical RAM since Linux decompresses it into a tmpfs.
Or if you set rootfstype=ramfs, then you can take up to all of physical RAM, but ramfs isn't swappable.
Several mentions here of using overlay file-system via the RaspberryPi specific raspi-config options.
However, where-ever systemd is the init one can do it via adding to the kernel command line
systemd.volatile=overlay
"If this setting is set to "overlay" the root file system is set up as "overlayfs" mount combining the read-only root directory with a writable "tmpfs", so that no
modifications are made to disk, but the file system may be modified nonetheless with all changes being lost at reboot."
See man 7 kernel-command-line for the other volatile mode options.
The Pi I have powering https://hiandrewquinn.github.io/selkouutiset-archive/ has a read-only / . Everything happens in RAM, and resetting everything back to normal is a matter of unplugging and replugging the device back in.
I could do so much more with this resource, but I'm honestly quite happy with it as it is.
You can skip the ntp/systemd-timesyncd/fake-hwclock stuff if you add a RTC/coin battery to the device. The Raspberry Pi 5 has built-in RTC with support for an official battery which costs about five bucks and works well for this. Older devices have aftermarket RTC's with battery backups as well.
These Pi* devices have served well running their Debian version for years.
I had a Pi3b run ADS-B (generally RTL-SDR) for many years. Dang though, all with (seemingly) random SD storage failures. Article details seems like it can help.
There are more options now though. Aarch64 ones that use eMMC.
I replaced the Pi3b with a Pine64 Rock64 running FreeBSD 14 booting from eMMC. Simple and just works with all RTL-SDR needs covered.
Readit News
Both SquashFS and EROFS are filesystem specifically designed for this kind of embedded, read-only use case. The former is optimized for high data density and compression, and already well established. The later is comparatively new and optimized for high read speed.[3] SquashFS as a rootfs can already be found in many embedded applications using flash storage and is typically also combined with tmpfs and persistent storage mount points or overlay mounts.
For both those filesystems, one would build a rootfs image offline. In the Debian ecosystem, there already exists a tool that can bootstrap a Debian image into SquashFS[4].
[1] https://en.wikipedia.org/wiki/SquashFS
[2] https://en.wikipedia.org/wiki/EROFS
[3] https://www.sigma-star.at/blog/2022/07/squashfs-erofs/
[4] https://manpages.debian.org/testing/mmdebstrap/mmdebstrap.1....
I can't vouch for every single distribution out there, but I've done this successfully on Debian at least.
I also have a pi running pikvm. You change the filesystem by doing:
- Use a power supply that can deliver enough current
- Use good USB cables that can actually carry said current (a lot of people miss this one)
- Do not use cheap SD cards
- Oversize your card so that the card can properly perform wear-levelling.
I have a pair of Raspberry Pi's (2's and/or 3's IIRC) running constantly since 2015. They're running an ancient version of Redsleeve (and are on my list to rebuild but never mind that). DNS, DHCP, and a few other services run on them. They've also survived multiple abrupt powerfail events.
They log to the SD card, and yet I've never had a problem. I personally believe that the fact that I'm using oversized cards with gigabytes of free space is the key. Well, perhaps that and the fact that my personal home directory is on NFS.
Or maybe I'm just lucky? Who knows...
So you reduce the chance of SD failure but introduce a risk of network failure. Of course, network failure doesn't have to be fatal but you won't have any logs to know what happened.
In my experience the chance of one of my Pi's not being able to access the network for one reason or another is much larger than a failed SD card, so I keep my logs on the card.
The Pis are energy efficient and worth running apps where catastrophic data loss aren't a concern. I use my Pi 1 with DietPi to run AdGuard and my Pi 4 for simple web apps, like dashboards.
But the world has moved forward and there are options like the ODroid that are almost as cheap and energy efficient, without the relying on an SD card. I run my Home Assistant off my ODroid with nvme. They are all backed up to a couple different places, but my failure expectations are completely different. I fully expect my Pis to just have completely ruined cards one day, whereas I am more worried about a bad update to my Home Assistant server.
I think this is half of the problem: SDHC does not require wear-leveling at all, I am unsure of SDXC. But regardless, the wear-levelling algorithm the SD card firmware does is completely invisible, from some vendors it might do a decent job, from some it might be a no-op, we have no way to tell.
The only way to know you're getting wear-levelling is to do it at an OS level, e.g. UBIFS stacked on MTD emulation - but there's a bunch of assumptions there which make it unsuitable for SD cards.
Yes, but then the budget then quickly rises to a point where a compact desktop such as a used Intel NUC is cheaper, on top of being far more powerful.
Got a USB cable tester and found both had cables with 1+ Ohms resistance, which would cause drop to less than 4V under load!
Swapped both cables to some good ones and never had an issue since.
I always use A1 or ideally A2 rated cards from reputable brands bought from reputable stores. Not sure if they help with longevity but the additional IOPS are very noticeable.
for stuff that has to be more reliable, I use openwrt (pi + ntp + usb gps + rtc => a standalone ntp server)
Or maybe I'm just lucky also.
For my own deployments, I ended up doing read only boot from SD card, making use of a ram disk for temporary files, and hosting the actual os on a USB stick, which was much better shielded from power issues. Worked great, never had issues, but the setup was pretty non standard.
It's just a few lines of code in initramfs!
and `update-initramfs -u`.Or if you set rootfstype=ramfs, then you can take up to all of physical RAM, but ramfs isn't swappable.
Use an USB Flash drive mounted as a R/W filesystem for logs, etc. fsck can usually recover those f/s when crashed. The root f/s remains intact.
However, where-ever systemd is the init one can do it via adding to the kernel command line
"If this setting is set to "overlay" the root file system is set up as "overlayfs" mount combining the read-only root directory with a writable "tmpfs", so that no modifications are made to disk, but the file system may be modified nonetheless with all changes being lost at reboot."See man 7 kernel-command-line for the other volatile mode options.
I could do so much more with this resource, but I'm honestly quite happy with it as it is.
I had a Pi3b run ADS-B (generally RTL-SDR) for many years. Dang though, all with (seemingly) random SD storage failures. Article details seems like it can help.
There are more options now though. Aarch64 ones that use eMMC.
I replaced the Pi3b with a Pine64 Rock64 running FreeBSD 14 booting from eMMC. Simple and just works with all RTL-SDR needs covered.
It's great to have options, both hw and os.