News for hackers. Stuff that matters.
Re:What’s New in Apple Filesystems [pdf] by bayindirh on Tuesday June 11, 2019 @04:37PM (#20159288)
Yes, that's a possibility. Flash level checksum only guarantees that the written or read LBA is intact.

However, I've never seen an enterprise storage system to do a mistake like that. My archaic OCZ Vectors or Mac SSDs also didn't do something similar. The problem you mention is as sinister as bit-rot.

It's always nice that the FS is self aware about its integrity, however it may be overkill for consumer class devices and installations. Maybe making the flash controller and the storage itself more reliable should be better in the long run for these type of devices IMHO.

Re:What’s New in Apple Filesystems [pdf] by monocasa on Tuesday June 11, 2019 @05:37PM (#20159755)
Flash level checksums also generally cover the flash translation layer (ie. the mapping of physical to logical sectors).

It's probably not needed for a vertically integrated stack of APFS on a Apple SSD controller.

It'd be nice to have fir external drives though.

Re:What’s New in Apple Filesystems [pdf] by throw0101a on Wednesday June 12, 2019 @10:45AM (#20165162)
> It's probably not needed for a vertically integrated stack of APFS on a Apple SSD controller.

Apple code has no bugs that could send back bad data? Apple has access to hardware that does not experience bit rot? Apple's gear is not susceptible to firmware hacking?

* https://www.wired.com/2015/02/nsa-firmware-hacking/

The various old school SAN vendors have all sorts of control for storage controllers and disk firmware, and I've seen ZFS checksum errors appear on scrubs with them.

Re:What’s New in Apple Filesystems [pdf] by monocasa on Wednesday June 12, 2019 @11:11AM (#20165402)
> Apple code has no bugs that could send back bad data? Apple has access to hardware that does not experience bit rot? Apple's gear is not susceptible to firmware hacking?

'It' being checksums over data that they already know is checksummed by the F2L layer and they know the semantics of.

If you're hacking the firmware, you know that APFS is running on top of it, so you would just fix-up it's checksums too. APFS having checksums doesn't get you anything there either.

> The various old school SAN vendors have all sorts of control for storage controllers and disk firmware, and I've seen ZFS checksum errors appear on scrubs with them.

You have next to zero introspection into how those drives actually work, or where the semantics of their F2L breaks down the easiest (they hide all of that out of patent concerns). For instance do those drives work around a write ahead log primarily, or is it a more traditional block renaming tree? How many generations are there? All the BS counters in the world are next to meaningless without some base implementation knowledge that's missing.

Re:What’s New in Apple Filesystems [pdf] by zzzcpan on Wednesday June 12, 2019 @11:36AM (#20165710)
Say you have a bug that corrupts data which somehow happens after writes so filesystem level checksums can detect it and you run that filesystem on desktop/laptop/mobile without redundancy. Does it actually help you with anything since you can't correct that error on the same filesystem level? Of course not. Does it at least prevent inconsistency? No, it doesn't unless failed checksums stop filesystem from doing new reads and writes. So the only sensible place for checksumming is on the level where such errors can be automatically corrected. Without redundancy that's just your disk controller or applications designed with data recovery in mind that also can do checksumming, not filesystems though.
Re:What’s New in Apple Filesystems [pdf] by bayindirh on Wednesday June 12, 2019 @07:36PM (#20170451)
Data doesn't fly from CPU to disk. It has to pass the RAM, storage controller, some caches most of the time and then it reaches to the medium that stores the data.

Data can be corrupted in anywhere in this chain. Checksum or block can end up in a bad memory cell, your storage controller can bit-flip or the universe can corrupt your data on the bus. Larger storage systems also add more connections, controllers, caches and disks. So the probability increases. This is why your flash, RAID controller, transmission protocols, HDDs, RAM and everything in between have error correction capabilities in enterprise systems. However consumer systems lack some of these.

Also, not all data ends up in a cutting edge SSD. HDDs are still present and being produced. While HDDs also have some block level error correction, they are susceptible to bit rot.

At the end of the day, FS level checksumming is desirable but, it costs space and computation. Instead of reducing the runtime and performance of the computer, consumer devices rely on the recovery mechanisms of the hardware and prevent most of the problems most of the times with good-enough resiliency and reliability.

Re:What’s New in Apple Filesystems [pdf] by zzzcpan on Wednesday June 12, 2019 @08:21PM (#20170730)
> Data can be corrupted in anywhere in this chain.

Of course, but most of this kind of corruption is trivial to recover from with just retries, that's why checksumming there works, but not on the filesystem level, where it can only detect corruption. What makes a bit more sense is something like erasure coding. Filesystem with erasure coding would probably be useful for all the broken FTLs on SD cards, eMMCs, SSDs, but at the end of the day wouldn't provide any significant level of resilience for a single disk to be worth the effort and the overhead. And checksumming alone still won't be useful.

Re:What’s New in Apple Filesystems [pdf] by throw0101a on Thursday June 13, 2019 @09:15AM (#20174499)
> Of course, but most of this kind of corruption is trivial to recover from with just retries, that's why checksumming there works, but not on the filesystem level, where it can only detect corruption.

FS-level checksum saves you from the situation when the underlying storage thinks everything is okay but it is not:

> This means there was a data error on the drive. But it’s worse than a typical data error — this is an error that was not detected by the hardware.

* http://changelog.complete.org/archives/9769-silent-data-corr...

* https://research.cs.wisc.edu/wind/Publications/zfs-corruptio...

* https://news.ycombinator.com/item?id=13851349 (via)

It's not an either-or situation, but rather both-and hardware- and FS-level checks.

Re:What’s New in Apple Filesystems [pdf] by zzzcpan on Thursday June 13, 2019 @09:29AM (#20174625)
> FS-level checksum saves you from the situation when the underlying storage thinks everything is okay but it is not:

It doesn't save you from anything, it just turns bit flips into full blocks of data loss and introduces overhead. If your application can deal with data loss it can deal with bit flips too. Filesystem-level checksumming is still useless.

Re:What’s New in Apple Filesystems [pdf] by throw0101a on Thursday June 13, 2019 @09:48AM (#20174785)
It tells you that your data is corrupted, which gives you options on dealing with the situation.

> If your application can deal with data loss it can deal with bit flips too.

No, it may not. The file may mostly be okay except for a place where a single bit flip causes the value to go from positive to negative:

* https://en.wikipedia.org/wiki/Signed_number_representations

Hardware not-handling bit flips is a documented phenomenon per my citations above. It does cause issues further up the stack. FS-level checksumming gives people the option to at least detect these errors, and depending on the situation, also perhaps fix them.

Re:What’s New in Apple Filesystems [pdf] by zzzcpan on Thursday June 13, 2019 @10:03AM (#20174939)
People already have a much better option by doing checksumming if necessary on the application level, much closer to the source of data actually with even less possibilities to miss corruption.
Re:What’s New in Apple Filesystems [pdf] by throw0101a on Thursday June 13, 2019 @10:19AM (#20175104)
Yes, doing it at the application level would be better. But doesn't preclude it doing it at other layers as well.

It's why we have Ethernet checksums, IP checksums, TCP/UDP checksums, etc. Defense in depth.

Re:What’s New in Apple Filesystems [pdf] by monocasa on Thursday June 13, 2019 @02:48PM (#20177580)
We don't have IP checksums, and only have TCP and UDP checksums because it's optional on Ethernet, and TCP/UDP doesn't always run over Ethernet anyway.
Re:What’s New in Apple Filesystems [pdf] by throw0101a on Friday June 14, 2019 @08:12AM (#20182307)
Re:What’s New in Apple Filesystems [pdf] by vardump on Thursday June 13, 2019 @02:15AM (#20172240)
> Of course, but most of this kind of corruption is trivial to recover from with just retries...

Retries won't help you when the data was corrupted on its way to be stored.

Re:What’s New in Apple Filesystems [pdf] by zzzcpan on Thursday June 13, 2019 @09:15AM (#20174496)
> Retries won't help you when the data was corrupted on its way to be stored.

Why not? This is exactly how such corruption is traditionally fixed. On crc error data is rejected before it is stored and retry then fixes the problem. And it's not that big of a deal either, only applications can get data checksummed in the earliest possible moment anyway. The hard part is automatically recovering from data loss and corruption later, which also makes it even unnecessary to verify if data is stored without corruption.

Re:What’s New in Apple Filesystems [pdf] by throw0101a on Thursday June 13, 2019 @09:23AM (#20174565)
> > Retries won't help you when the data was corrupted on its way to be stored.

> Why not?

1. Data is fine in memory

2. Data is fine over PCIe.

3. Data is corrupted by firmware bug in storage controller.

4. Data (now corrupted) is sent to disk, which generates checksums for each LBA, and sends the bits to the storage medium.

5. Some time later the data is requested, disk reads bits from sectors, checksums pass, corrupted data is sent back up the stack to the application.

This is not a fantasy scenario:

* http://www.cs.toronto.edu/~bianca/papers/fast08.pdf

* https://storagemojo.com/2007/09/19/cerns-data-corruption-res...

Re:What’s New in Apple Filesystems [pdf] by zzzcpan on Thursday June 13, 2019 @09:42AM (#20174733)
You are missing the point: checksums can't save you from data corruption and data loss. They are unnecessary without automatic recovery mechanisms and even make things worse.
Re:What’s New in Apple Filesystems [pdf] by throw0101a on Thursday June 13, 2019 @09:59AM (#20174898)
> You are missing the point: checksums can't save you from data corruption and data loss.

No, I am not. The point is: checksums allow you to know that data corruption happened in the first place. Knowing this, you can then make informed decisions.

What saves you from "data corruption and data loss" is backups. But how do you know you have to go to backups if you're not aware that your data has problems?

> They are unnecessary without automatic recovery mechanisms and even make things worse.

Checksums allow you do know that there is a problem in the first place. If you don't know there's a problem, how can you invoke "recovery mechanisms", automatic or otherwise?

If you use (say) ZFS with some level of RAID-Z over multiple disks, then you get an "automatic recover mechanism". But even for a single-disk situation, it would be handy (for me at least), to know a file is bad so I can restore it from (say) my Time Machine backup.

But I have to know that the file is corrupted in the first place. And checksums would give me that.

Re:What’s New in Apple Filesystems [pdf] by zzzcpan on Thursday June 13, 2019 @10:19AM (#20175099)
> What saves you from "data corruption and data loss" is backups. But how do you know you have to go to backups if you're not aware that your data has problems?

It doesn't actually. It introduces inconsistencies unless on first detected corruption your filesystem stops all reads and writes until you recover corrupted block from backup and you might not even have it in backup, because backup is not synchronous replication.

> Checksums allow you do know that there is a problem in the first place. If you don't know there's a problem, how can you invoke "recovery mechanisms", automatic or otherwise?

You do your own appropriate checksumming suitable for recovery mechanism that you have. It's hardly as primitive as you think, because you have to respond fast, without corruption, transparently on request, you also have to schedule a check whether you actually need to heal anything, maybe it was a transient error, then mark it as corrupted, find what to resync from where, all of which also has to be fast, which means a bit complicated algorithms with sort of checksum databases on each replica, etc.

Re:What’s New in Apple Filesystems [pdf] by throw0101a on Thursday June 13, 2019 @10:25AM (#20175153)
What do you do as a living? Because I think we are living in different realities.

I, as a sysadmin, have been helped by (ZFS) checksums multiple times. This includes times where I had to go to tape manually because the really expensive SAN has said "everything is fine" when it wasn't.

So telling me "checksums do not help" is contrary to my life experience.

Re:What’s New in Apple Filesystems [pdf] by zzzcpan on Thursday June 13, 2019 @10:31AM (#20175212)
I mostly speak from experience designing, implementing and operating a distributed database/storage system and traditional storages, databases before that.
Re:What’s New in Apple Filesystems [pdf] by throw0101a on Thursday June 13, 2019 @10:39AM (#20175301)
And you've never seen the storage stack give back bad data?

You seem to be saying "checksums don't matter because there may be nothing we can do about about the corrupted data". So we should just blindly accept all the bits from the lower levels of the stack? We should do nothing to verify that those levels of the stack are working correctly?

You've mentioned application-level checksumming: yes, that would be good. But until every single dev adds code to checksum (and correct?) their data structures and file formats, having more checksums allows for beter information about when corruption occurs IMHO.

Re:What’s New in Apple Filesystems [pdf] by zzzcpan on Thursday June 13, 2019 @11:00AM (#20175524)
My point is checksumming is not a solution for data corruption, never was, never will be. It's only a tiny part of more complex algorithms and systems that detect and fix corrupted data automatically, never something usable on its own.
Re:What’s New in Apple Filesystems [pdf] by throw0101a on Thursday June 13, 2019 @11:44AM (#20175960)
Checksumming is necessary, but not sufficient, for proper data integrity. But if you don't have it, you can't do anything else.

So why does it seem you are agruing against checksums?

Re:What’s New in Apple Filesystems [pdf] by throw0101a on Thursday June 13, 2019 @09:06AM (#20174407)
> Does it actually help you with anything since you can't correct that error on the same filesystem level? Of course not.

Given that I run Time Machine, yes it does: it tells me I have to restore the file from a backup.

I also use SuperDuper to create bootable clones on a rotating set of drives that I take to work for offsite backup, so the file would probably be there as well. (Though this is a bit extreme for a non-geek to do.)

Re:What’s New in Apple Filesystems [pdf] by bayindirh on Wednesday June 12, 2019 @02:04AM (#20162135)
Flash level checksums are an integral part of wear-leveling and flash block allocation strategy. So they are also used for block migration/retirement controls. In other words, it's not trivial or desirable to remove flash level ECC or other checksums from physical flash management layer.

I cannot find the article now, but the flash storage is way less reliable than it seems and it's correcting bit errors silently without reporting most of the time.

This article [0] shows some of the things behind the curtain in flash storage configuration and lifetime. I'd edit or add the other article if I can find it.

[0]: https://goughlui.com/2015/04/05/teardown-optimization-comsol...

Re:What’s New in Apple Filesystems [pdf] by monocasa on Wednesday June 12, 2019 @11:00AM (#20165309)
I'm saying that flash level checksums on a vertically integrated stack make checksumming again at the FS layer moot, assuming you have proper control and reporting on both sides of the block device abstraction boundary. Apple's about the only one who can pull this off with their vertically integrated stack.
Re:What’s New in Apple Filesystems [pdf] by bayindirh on Wednesday June 12, 2019 @07:23PM (#20170372)
> ...flash level checksums on a vertically integrated stack make checksumming again at the FS layer moot...

This is simply not true. First, flash is not always the place data ends at. Flash is prevalent in more points in the storage chain. Predominantly on RAID controllers' caches in last 3-5 years and hybrid drives used in laptops. So, flash-level checksumming can guarantee that the data is retrieved in the form it's written and allows neat tricks like wear-leveling and such. On the other hand data can get corrupted in other places.

e.g. Data can get corrupted in RAM, bit-rot in hard drives, RAID controllers can add a thick layer of complexity which you cannot manage or the filesystem can just make mistakes due to bugs (I've seen it all). So flash-level checksum can protect the data on the flash, but cannot guarantee that the data is correct in the first place. So if you checksummed the data and wrote it differently, FS or RAID level checksumming will catch it. This will prevent domino effects which the corrupted data can create.

Data management gets complicated fast when you need more than a few disk's aggregate capacity with high performance and reliability and, FS level checksumming is a powerful tool to ensure that.

> ...assuming you have proper control and reporting on both sides of the block device abstraction boundary.

Again this will only guarantee that the device reads the data it has written, but it won't guarantee that it had received the correct data in the first place.

> Apple's about the only one who can pull this off with their vertically integrated stack.

While Apple can do a lot of exotic stuff with the hardware, it won't be wise to create a custom flash controller and protocol to fuse FS and flash level checksums. It'll increase the software and integration cost, limit their options in hardware scene, slow down product development and plainly limit them in the long run.

I had an Intel 4 channel RAID card which corrupted the array when it wakes up from the wrong side of the bed and, it spent hours to verify and rebuild the array from the block level checksums it created on the disks during array creation. I've never lost data, but decided that it didn't worth the effort to keep the array alive and well. So, I eventually ditched it. However its activity LEDs on the board was kinda nice.

Re:What’s New in Apple Filesystems [pdf] by monocasa on Thursday June 13, 2019 @02:35PM (#20177469)
> While Apple can do a lot of exotic stuff with the hardware, it won't be wise to create a custom flash controller and protocol to fuse FS and flash level checksums. It'll increase the software and integration cost, limit their options in hardware scene, slow down product development and plainly limit them in the long run.

I mean, tell that to them, because that's what they've done. The T2 chip's feature set is very married to APFS. AFAIK, Linux still doesn't have working drivers for it despite it sort of looking like an NVMe drive.

Re:What’s New in Apple Filesystems [pdf] by zzzcpan on Wednesday June 12, 2019 @11:38AM (#20165741)
Anyone can pull this off, because filesystem level checksumming without an additional place to recover data from is useless.
Re:What’s New in Apple Filesystems [pdf] by bayindirh on Wednesday June 12, 2019 @07:38PM (#20170468)
Exchange the checksumming with hamming codes or with another ECC system with >=2 bit detection/correction capability and you're all set.
Re:What’s New in Apple Filesystems [pdf] by zzzcpan on Wednesday June 12, 2019 @08:29PM (#20170782)
Not really set, but even assuming you can make it work this is already way beyond checksumming the lack of which people complain about for some reason.
Re:What’s New in Apple Filesystems [pdf] by bayindirh on Thursday June 13, 2019 @07:02AM (#20173551)
Most of the big storage vendors are making it work with Erasure or Hamming Codes. They compensate damaged or missing disks in the arrays transparently and send you a small e-mail about the details of the disk you need to change.

RAID-5 and RAID-6 is the most primitive and probably most deployed version of this checksum based compensation and recovery methods.

Erasure/Hamming is used for bigger arrays where RAID-5/6 cannot build the disks in a reasonable amount time while they are under I/O load.

Re:What’s New in Apple Filesystems [pdf] by monocasa on Thursday June 13, 2019 @02:36PM (#20177481)
Pretty much every NAND flash device out there uses full ECC rather than just a checksum. The whole industry relies on it for yield reasons.
Re:What’s New in Apple Filesystems [pdf] by throw0101a on Thursday June 13, 2019 @10:10AM (#20175007)
> However, I've never seen an enterprise storage system to do a mistake like that.

Unless you're using a checksumming file system (a la ZFS), how would you know? And even if you were using ZFS, it would probably just do a re-read from another part of the mirror/RAID-Z and fix it behind the scenes.

That's the issue with people saying "things are fine": how do they actually know since we're not verifying things?

Up until fairly recently, we haven't really been looking too closely, simply trusting the lower levels of the stack:

* http://www.cs.toronto.edu/~bianca/papers/fast08.pdf