I second this however I don't use the filesystem to get this functionality. I most often use XFS and have a cronjob that calls an old perl script called "rsnapshot" [1] that makes use of hardlinks to deal with duplicate content and save space. One can create both local and remote snapshots. Similar to your situation I have used this to fix corrupted git repos which I could have done within git itself but rsnapshot was many times easier and I am lazy.
https://kb.synology.com/en-us/DSM/tutorial/What_was_the_RAID...
For scrubbing, plain-old btrfs-scrub is being used.
and they'd be correct
However, the checksum must be chosen at the time of filesystem creation. I don't know of any way to upgrade an existing BtrFS filesystem.
Contrast this to ZFS, which allows the checksum to be modified on a file-by-file basis.
Bitrot still happens with ECC.
Having filesystem level protection is good, but it its like going from 80% to 85% protected. That is because the most critical part remaining unprotected in a traditional RAID/etc system is actually the application to filesystem interface. Posix and linux are largly to blame here because the default IO model should be async interface where the completion only fires when the data is persisted and things like read()/write()/close() should be fully serialized with the persistence layer. Otherwise, even with btrfs the easiest way to lose data is simply write it to disk, close the file, and pull the power plug.
For example if your use case is a file archive (think raw photo or video), then fiesysytem interface does not matter - if the computer crashes soon after copy, you re-copy from original media. But bit flips are very real and can ruin your day.
But as I mentioned these days I'm pretty sure nearly all the storage loss that isn't physical damage is actually software bugs. Just a couple months ago I uploaded a multiple GB file (on my ECC protected workstation) to a major hyperscaler's cloud storage/sharing option. Sent the link to a colleague half way around the globe and they reported an unusual crash. So I asked them to md5sum the file and they got a different result from what I got, so I downloaded the file myself and diffed it against the original and right in the middle there was a ~50k block of garbage. Uploaded it again, and it was fine. Blame it on my browser, or whatever if you will, but the end result was quite disturbing because I'm fairly certain my local storage stack was fine. These days i'm really quite skeptical of "advanced" filesystem/etc. What I want is a dumb one where the number 1 priority is data consistency. I'm not sure that is an accurate reflection of many of them, where winning the storage perf benchmark, or feature wars seems to be a higher priority.
I never found the cause, because I just switched to a completely different system to copy data. I know it was not disk-specific because this was happening on multiple hard drives, nor it was physical damage, as SMART/syslog were silent, and reading the disk again was giving correct data. Memory was fine -- not ECC, but I did run a lot of memtest's on it.
Later on, I found some blog posts which mentioned the similar problem and claim it was result of bad SATA card, or bad cable, or even bad power supply. I remember there was original one made by Jeff Bonwick on his ZFS blog, but I cannot find it anymore. Here is a more modern link instead: https://changelog.complete.org/archives/9769-silent-data-cor...
I now have the homegrown checksumming solution which I use after each major file transfer, and I have not seen any data corruption yet (*known on the wood().
I think the answer is its all covered.
Looking at even SATA 1, if it is implemented _CORRECTLY_ you get the packet CRC protecting the link data from the point its formed to the point the endpoint verifies the result. So like ethernet sometimes it doesn't matter if some random piece of junk in the process doesn't do its own ECC validation, because its covered under a higher level of the stack.
If your adapters are "desktop" grade then I might consider seeking another vendor if you care about data integrity, some vendors are definitely shipping crap, but there are vendors I can assure you will detect link/etc failures.
And as a side, note, I've seen a lot of bad data, a huge percentage of it, were kernel/etc filesystem errors. We added a bunch of out of band extra metrics to track when/where writes were going and our own metadata layers/etc, and it uncovered a whole bunch of software errors.
>>it is designed with a focus on data integrity by protecting the user's data on disk against silent data corruption caused by data degradation, power surges (voltage spikes), bugs in disk firmware, phantom writes (the previous write did not make it to disk), misdirected reads/writes (the disk accesses the wrong block), DMA parity errors between the array and server memory or from the driver (since the checksum validates data inside the array), driver errors (data winds up in the wrong buffer inside the kernel), accidental overwrites (such as swapping to a live file system), etc.
Also tagging is better than writing it down. Like initials+date.
Don't insult people's intelligence then command them to spend hours to prove that you're wrong.
Just start with the assumption that you may be wrong next time.
Also I wouldn't trust BTRFS for the purpose of data archiving, for the fact that ext4 is a proven (and simpler) filesystem, thus it's less likely to became corrupt, and it's more likely to being able to recover data from it if it will become corrupt (or the disk has some bad sectors and that sort of stuff).
On the contrary; I'm using btrfs and not ext4 on NAS (Synology) specifically, because the former does checksumming and bitrot detection and the latter does not.
The backup I was referring to is the offline one. If I need to backup data, something that unfortunately I don't do as often as I should, I need a filesystem that is reliable, that is proven (ext4 is around since more than a decade, and if we count the previous version even more, so in 20 years I'm confident that I would be able to mount an ext4 hard drive that I forgot in the garage in a modern system, with BTRFS, who knows), and that for it they exist a lot of tools in case something goes wrong (there are ton of tools to recover data from damaged ext4 drives, are we sure that with BTRFS is as easy? If I have a filesystem with compression recovering data I don't think is a simple as running photorec...)
Also, the filesystem of a backup drive is not something you can change easily. I still have an old 1Tb drive that I formatted long long time ago in NTFS, and I never changed the filesystem since having to backup all the data to another drive (find it another 1Tb drive), format the drive, and copy back the data will take 1 day. Not that there are things super important on that drive, mostly is stuff I downloaded from the internet years ago, still it's an example why for a backup drive I don't want to have the cutting edge choice that then creates problems in the future.
Ext4 is ubiquitous, so it's my filesystem of choice for all the purpose that have the requirement that the data must be archived for more than 2 years.
Quite a lot of the assumptions of earlier file systems is the hardware either returns correctness, or reports a problem e.g. uncorrectable read error or media error. That's been shown to be untrue even with enterprise class hardware, largely by the ZFS developers, hence why it exists. And also why ZFS has had quite a lot less "bad press" where Btrfs wasn't developed in a kind of skunkworks, it was developed out in the open where quite a lot of early users were using it with ordinary every day hardware.
And as it turns out, we see most hardware by make/model doing mostly the right things, a small number of make/models, making up a significant minority of usage volume, don't do the right things. Hence, Btrfs has always had full checksumming of data and metadata. Both XFS and ext4 were running into the same kinds of problems Btrfs (and ZFS before it) revealed - torn writes, misdirected writes, bit rot, memory bit flips, and even SSD's exhibit prefail behavior by returning either zeros or garbage instead of data (or metadata). XFS and ext4 subsequently added metadata checksums, which further reinforced the understanding that devices sometimes do the wrong thing and also lie about it.
It is true that overwriting filesystems have a better chance of repairing metadata inconsistencies. A big reason why is locality. They have fixed locations on disk for different kinds of metadata, thus a lot of correct assumptions can be made about what should be in that location. Btrfs doesn't have that at all, it has very few fixed locations for metadata (pretty much just the super blocks). Since no assumptions can be made about what's been found in metadata areas, it's harder to fix.
So the strategy is different with Btrfs (and probably ZFS too since it has a fairly nascent fsck even compared to Btrfs's) - cheap and fast replication of data via snapshots and send/receive, which requires no deep traversal of either the source or destination. And equally cheap and fast restore (backwards replication) using the same method. Conversely, conventional backup and restore are meaningfully different when reversing, so you have to test both the backup and restore to really understand if your backup method is reliable. That's going to be your disaster go to rather than trying to fix them. Fixing is almost certainly going to take much longer than restoring. If you don't have current backups, at least Btrfs now has various rescue mount options to make the file system more tolerant of broken file systems, but as a consequence you also have to mount read-only. Pretty good chance you can still get your data out, even if it's inconvenient to have to wipe the file system and create a new one. It'll still be faster than mucking with repair.
Also, Btrfs since kernel 5.3 has both read time and write time tree checkers, that verify certain trees for consistency, not just blindly accepting checksums. Various problems are exposed and stopped before they can cause worse problems, and even helps find memory bitflips and btrfs bugs. Btrfs doesn't just complain about hardware related issues, it'll rat itself out if it's to blame for the problem - which at this point isn't happening any more often than ext4 or XFS in very large deployments (millions of instances).
I didn't talk only about corruption of the filesystem itself (I don't know if it's more or less likely with BTRFS, someone says that BTRFS is more likely to become corrupt with power failures, I don't know if it's true), but also from hardware failures. In case of a disk with damaged sectors (I know that we should have 3 backups with one offsite, but you always have that one disk with important data on it that it's a year that you are promising to backup next day till it breaks) I think that a filesystem with a simpler structure will lead to an higher probability of recovering the data, while I think that with BTRFS, or any filesystem that is COW, uses compression, volumes, etc that is more difficult, because files are not stored as plain blocks on the disk, but have a more complex structure that must need to be decoded.
Also BTRFS is kind of a new filesystem, that has two disadvantages, there are not all the tools that were developed over the years for ext4, and also BTRFS driver is continuing evolving. Why I can be pretty confident that if I format an hard disk today with an ext4 filesystem in 20 years I will find a driver for a modern Linux (or whatever OS will replace it in 20 years) to mount it, can we have the same assurance with BTRFS? I don't know.
So for the purpose of making backups and archiving data, I think that I will stick with ext4 for a while. While on my laptop, and systems that I use, I use BTRFS without any problems.
No. If the drive honors flush/FUA, Btrfs is less likely to corrupt data or metadata than overwriting file systems because the interruption won't result in incomplete overwrites. So this would hold true for any copy-on-write vs overwriting file system (and probably also log based file systems). The trouble is if the drive is transiently lying about flush/FUA success, and then there's an ill timed crash. There's the chance the super blocks written point to trees that don't exist yet because the write order hasn't been honored due to flush/FUA being ignored. There are backup trees, so it might be possible to work around this defect with the `rescue=usebackuproot` mount option, but sometimes the defect is so bad that you get all kinds of write reordering such that Btrfs only finds trees with the wrong generation, and it fails to mount. Often it's still possible to get your data out with the offline scrape tool, `btrfs restore`. But it's a difficult problem to deal with. In theory it's similar on ZFS but I know nothing about its on-disk format so maybe its metadata has some locality in which case certain assumptions could be made to allow it to better work around such a drive firmware defect? I'm not sure. On a power fail, it is possible Btrfs loses the most recently written data if the writes that were in-progress and thus not yet fully committed to stable media. How much data really depends on the application doing the writes.
>In case of a disk with damaged sectors
Btrfs by default keeps two copies of metadata and it automatically deals with this problem, while also self-healing when such problems are encountered.
>a filesystem with a simpler structure will lead to an higher probability of recovering the data, while I think that with BTRFS, or any filesystem that is COW, uses compression, volumes, etc that is more difficult, because files are not stored as plain blocks on the disk, but have a more complex structure that must need to be decoded.
The ondisk format is fairly simple and extendible. Metadata isn't subject to compression. In the case of bad sectors with compressed (user) data, you'll certainly lose more data than if it weren't compressed. There's an expected trade off here, it's not really a Btrfs issue but just the way all compression algorithms work. You get some small corruption and it'll have a bigger effect.
>So for the purpose of making backups and archiving data, I think that I will stick with ext4 for a while.
I used to hedge my bets by having multiple copies of data on different file systems (including ZFS) but haven't done that in years. I've seen too many cases of (hardware induced) data corruption being replicated into backups and archives without any warning it was happening until it was too late - and only corrupt copies remained.
BTRFS sounded cool with all it's new features, but the reality is that ext4+LVM does absolutely everything I need and it's never given me any issues.
I'm sure BTRFS is much more robust these days, but I'm still gun shy!
A week later, the power failed in the office, and my filesystem got corrupted. In the middle of repairing, the power dipped again, and my entire fs was unrecoverable after that. I managed to get the data off by booting off a separate drive and running a command to extract everything, but it would never mount no matter what I did.
I've never had an issue with ext4, xfs, or zfs no matter how much I've abused them over the past 10+ years, but if losing power twice can wipe out my filesystem then no thanks, I'm out.
(Plus: non-recursive snapshots? No thanks.)
EXT4 has never once failed me, and I personally battle tested it by working while losing power probably a total of 200 times. I probably should have bought a UPS come to think of it.
These were single disk systems, no raid at all.
Around the timeframe you mentioned, I lost a BTRFS filesystem when I filled it up. Probably could have recovered it if I had known more, but oh well. I definitely feel the gun-shyness!
However, I'd want to add that at a previous job, I had a ext4 fs go belly-up in a similar way. One day, just died without warning. Maybe could have recovered it, but like others have mentioned we'd have no guarantees about the data.
Moral of the story is, of course, always have backups :)
On the other hand, those days might be behind it. I haven't kept track.
The upstream maintainer of btrfs-progs also maintains btrfs maintenance https://github.com/kdave/btrfsmaintenance which is a set of scripts for doing various tasks, one of which is a periodic filtered balance focusing on data block groups. I don't run it myself, because well (a) I haven't run into such a bug in years myself (b) I want to run into such a bug so I can report it and get it fixed. The maintenance script basically papers over any remaining issues, for those folks who are more interested in avoiding issues than bug reporting (a completely reasonable position).
There's been a metric ton of work on ENOSPC bugs over the years, but a large pile were set free with the ticketed ENOSPC system quite a few years ago now (circa 2015? 2016?)
Does any other FS on Linux provide those?
But... any of the other stuff? Checksums, deduplication, compression, etc. Citations, please.
LVM VDO wasn't really a thing until a couple of years ago, and I've never actually heard anyone recommend using it. Definitely not within its "traditional domain".
Perhaps you mean on other operating systems... when this discussion is all about a Linux filesystem and whether other Linux filesystems provided these features, including the comment you replied to?
Yes. The Linux LVM logical volume manager for example can do those things.
Traditional domain meaning this kind of data management traditionally came under the purview of LVMs. Not that an implementation has had particular features for a length of time, but that you might (also) look there rather than the filesystem for those features.
As fair as I know the only solution for deduplication in LVM is VDO and that was created after ZFS and isn't technically part of LVM (it's a kernel module that sits between the filesystem and LVM).
Same is true for compression except in the case of compression that's been a file system feature since the days of NTFS on NT 3.5 (released 1995) -- I can't recall when UNIX file systems first saw compression but I'd wager it was before ZFS, which we've already acknowledge predates VDO.
I also thing it makes sense that the above should be part of the file system domain because it's an integral part to how the file system organizes the data (much like how we don't argue that the journal isn't part of the file system domain).
As for checksums, they have existed in all parts of the stack, from the application level (for example rsync), to the volume level (RAID controller cards) to the file system. I wouldn't say that's something that has ever had a "traditional domain".
The things I'd say ZFS and its ilk do that isn't part of its traditional domain is RAIDing (ie management of the physical devices) and cache control. Both make a lot of sense being managed by the file system in modern file systems but I can also sympathize with those who think that's a step too far. In the case of RAIDing, you can still run ZFS on LVM or a hardware controller if you wish -- I wouldn't suggest you do so because it gives you additional complications for zero benefit but the option is still there if you want it. But with regards to cache, the only way to opt out of ZFS managing it's own cache is not to use ZFS.
Anyway it wasn't really a statement about what was there first or not, but rather that because something may not exist for a filesystem does not mean it does not exist, i.e., look at other layers for such functionality.
https://www.redhat.com/en/blog/look-vdo-new-linux-compressio...
One can also try convert a desktop installation into a server installation (not sure if it’s possible)
That said, I'm thinking about leaving a BTRFS partition unmounted and mounting it only to perform backups, taking advantage of the snapshotting features.
Device replace
--------------
>Device replace and device delete insist on being able to read or reconstruct all data. If any read fails due to an IO error, the delete/replace operation is aborted and the administrator must remove or replace the damaged data before trying again.
Device replace isn't something where "mostly ok" is a good enough status.
I mean I'm all for choice on Linux but when it comes to file systems I'd rather have fewer choices but those choices be absolutely rock solid. Btrfs might have gotten to that stage now however it's eaten enough peoples data (including mine) over the years that I can't help wondering why people bothered to persist with using it when a better option was available.
Try to fill your root-filesystem with dd, then remove the file and sync, reboot and enjoy a non booting OS ;) It's like they don't test it at all.
But then they make changes that add an insane amount of complexity, and suddenly you're running into random errors and googling all the time to try to find the magical fixes to all the problems you didn't have before.
I can safely say it has not presented any problem with me thus far, and I am at the stage of my life where I realize that I don't have the time to fiddle as much with settings. If the distributions are willing to take that maintenance on their shoulders, I'm willing to trust them and deal with the consequences – at least I know I'm not alone.