> Not to take away from the merits of ZFS, but this sounds like something that 2 disks with software RAID1 would prevent from happening.

You're not wrong. And if there's a mechanical issue with a drive and it dies / stops spinning, then you'll probably get an alert and can swap it.

But if there's any kind of bit rot or data corruption, and it only happens on one side of the (traditional) RAID1 mirror, how will you (a) know that it actually occurred, or (b) know which side is good bits and which side has bad bits?

With ZFS and checksums, you can be confident that your data is still healthy. Depending on the data, this may be an important consideration.

zfs send-recv is also very handy for doing incremental backups.

What ZFS does here is not enough for me. I need to be alerted to changes done also above the file system layer, like by malware or accidental deletion etc. Only way I have found to solve that is to have checksum tool above the file system. If something is wrong, I restore from backup, so zfs does not give me anything...

I don't personally do this, but if you're taking scheduled snapshots of your ZFS filesystems, you could also have a scheduled job (say, nightly) that emails you a "zfs diff" [0] between the current snapshot and the one from 24 hours ago. It won't tell you that you've been hit by malware, but an unexpected spike in changes could be something worth investigating.

[0] - https://docs.oracle.com/cd/E36784_01/html/E36835/gkkqz.html

> I need to be alerted to changes done also above the file system layer [...]

As others have mentioned, would "zfs diff ..." be useful?

* https://www.thegeekdiary.com/how-to-identify-zfs-snapshot-di...

As the name suggests, "snapshots" are read-only and so cannot be altered. You could either copy/rsync the modified file/s to the live location, or do a rollback to a particular snapshot:

* https://www.thegeekdiary.com/solaris-zfs-how-to-create-renam...

If the machine is compromised in some way, you could reinstall and do a "zfs send-recv" of a pool from a remote system.

You can clone the snapshot (so its clone become writable, not the snapshot itself!) and you even can promote a snapshot to a "parent" filesystem (reverse parent-child).

You can do a zfs diff command on two snapshots or a snapshot and current dataset as zfs takes a just-in-time snapshot of the live dataset.

One pro of ZFS over RAID1 is that ZFS is file based and not volume based, so in case of recreating mirror disks (or general resilvering) you only copy data, not the whole volume.

More than half the time in my experience it’s usually a failing sata cable or power-supply if the hard drive continues to functions on boot but eventually goes to degraded due to checksum/read errors. Sometimes the data is just fine but a read error occurs between the drive and controller which ZFS interprets as a failng disk

modern drives have onboard diagnostics which can report errors directly to the OS. I'd be very surprised if a modern file system was unaware of this. I'd also be surprised if users of alternative file systems didn't test a suspected failing drive before concluding it was faulty

Hard drives lie. SMART data is not thorough and it is not trustworthy. Across dozens of drive failures over the years, I've never seen SMART predict a failure before ZFS detected data loss or the OS reported 'drive is slow'. SMART is not worth the effort to monitor.

Filesystems don't read SMART data. You might have a separate daemon which monitors SMART.

ZFS checksumming is amazing for this. You know, without doubt, which file(s) are bad. You can still use failing or unknown quality drives because the checksumming will protect you from silent data corruption.

SMART marking drives as failed is usually super late. But if you monitor the raw values for the important parameters (unreadable, reallocated, etc sectors), I've had good luck with replacing drives before software notices. At least for spinning drives; SSD failues were way more rare, but resulted in the drives completely disappearing from the bus, and no reliable prefail indicators. I did have one SSD go through a big reallocation that tanked throughput and the alerts from throughput and the alerts from SMART thresholds fired simultaneously.

Of course, that's great for a server farm; in home use monitoring is a lot less structured.

Expensive enterprise drives are generally better at smart stats. Of course your mileage varies by vendor. Desktop drives on the other hand are completely untrustworthy.

the great thing about zfs is you don't have to care, at all

as long as it writes and reads back (most of the time): ZFS will deal with it

The error is occurring between the drive and the sata controller on the mobo. There’s also the whole issue of whether the OS or the drive is responsible for error handling. If the the drives are SAS drives, typically the OS uses fire and forget methodologies for write commands. Other drives might not support that and require zfs to handle errors at the software/kernel level

replacing the cable / testing the disk on an isolated system would confirm this

It would but I encountered a situation once, where when I would remove the drives of the array and test the disk 1 by 1 they would all work. It wasn't until all the drives were powered on and under max write stress would the power-supply under-volt to drives randomly making it appear as multiple drives were all failing.