Re:Apple’s big test of data integrity
by jeffbee on Tuesday May 30, 2023 @05:27PM
(#36131101)
How do they checksum 9GB of data and still boot promptly?
[ Parent
| on HN ]
FTA: "This is the ingenious part: verification carries little overhead, as it runs as a rolling process once the top-level ‘seal’ has been verified. So verification is a continuous process as the SSV is mounted and accessed."
Re:Apple’s big test of data integrity
by vlovich123 on Tuesday May 30, 2023 @07:03PM
(#36132336)
Which presumably means Merkel Trees.
Merkle trees! You (simplified) hash the data at the leaves of the tree, merge adjacent leaves (concatenation for example), and hash the new leaves, until you're left with just the Merkle Root, a single hash representing the entirety of your data. Verifying the root is easier than loading the data itself into memory and hashing it there since you can verify it piecemeal and without loading it all into memory at once.
But regardless of the ordering which you read, don't you ultimately still need to read every bit once?
Re:Apple’s big test of data integrity
by CJefferson on Tuesday May 30, 2023 @07:48PM
(#36132751)
Each level of the tree contains a hash of the hashes from the level below, so to verify the top hash you need only hash all its children (which are hashes themselves). Then you can explore a child, continuing recursively. At the bottom you find a hash of an actual file (or part of a file), then hash the file to check it’s validity.
This allows you to only hash those parts of the tree you actually want to read.
> Big Sur, Monterey or Ventura in default Full Security mode has verified every last bit of their 9 GB SSV.
In this case they're not selective, so the tree approach doesn't save the reads.
Re:Apple’s big test of data integrity
by vlovich123 on Tuesday May 30, 2023 @11:23PM
(#36134136)
If you want to verify the entire image, I don’t think you can get around reading the entire image. Because any part you didn’t read to verify the hash is a part that could contain corruption.
That's around 3s, for their slowest offering.
Don’t these machines have read speeds of like 5000 MB/s? So just read the 9 GB blockwise and apply a super simple hash. Should be done in two seconds.
If I include the fs overhead, I can shasum a single 23GB file (likely not fragmented) in 46s.
That's uncached, so it's not that quick at boot. 9GB would take ~19s then. Block would go faster, but not an order of magnitude faster.
Single threaded or in parallel?
It's a Merkle tree, apparently, so parallelizing the checksum should be trivial.