News for hackers. Stuff that matters.
Re:No, ZFS really doesn't need an fsck tool by ChuckMcM on Friday March 29, 2013 @11:31AM (#5461216)
Interesting rant (from 2009). At NetApp the WAFL file system is also always consistent on disk, so it too doesn't need fsck. That said, WAFL had 'wack' (WAfl ChecK) which could go through and check that the on disk image was correct.

Unlike UFS or FFS or EXTn the file system couldn't be corrupted by loss of power mid write, but like ZFS it can be corrupted by bugs in the code which write a corrupted version to disk. So the tool does something similar to fsck but it is simpler, more of a data structure check rather than a "recreate the flow of buffers through the buffer cache to save as much as possible" exercise.

Re:No, ZFS really doesn't need an fsck tool by gnosis on Friday March 29, 2013 @01:09PM (#5461698)
"At NetApp the WAFL file system is also always consistent on disk, so it too doesn't need fsck."

How does it manage to stay consistent if a cosmic ray strikes it and flips one or more bits?

How does it manage to stay consistent if you physically bump in to the drives and cause physical damage by having the disk head briefly touch the disk surface?

Wouldn't you need a filesystem consistency check and repair tool like fsck in these cases?

Re:No, ZFS really doesn't need an fsck tool by ChuckMcM on Friday March 29, 2013 @02:24PM (#5462176)
"How does it manage to stay consistent if a cosmic ray strikes it and flips one or more bits?"

At the time (and I think its still true) cosmic rays do not have sufficient energy to flip a magnetic domain on disk. Memory bit flips are detected by ECC and channel (between the I/O card and memory and/or disk) are identified with CRC codes.

"How does it manage to stay consistent if you physically bump in to the drives and cause physical damage by having the disk head briefly touch the disk surface?"

The disks are part of a RAID4 or 6 group (RAID 6 preferred for drives > 500MB, required for drives >= 2TB) so physically damaging a drive results in a group reconstruction of the data on that drive.

NetApp has always had a pretty solid "don't trust anything" sort of mantra that has been tested and fortified a few times by various events. The ones I got to see first hand were an HBA that corrupted traffic through it in flight, drives that returned a different block than you asked for, and drives that acknowledged they had written data to the drive when in fact they had not.

Back in the early 2000's anything that could happen with a disk with a probability larger than once in billion operations or higher, they got to see once a month. It was an interesting challenge which requires a certain discipline to deal with. When I went to Google and saw their "we assume everything is crap, we just fix it in software" model it gave me another perspective on how to tackle the problem of storage reliability.

Both schemes work and have their plusses and minuses.

Re:No, ZFS really doesn't need an fsck tool by X-Istence on Friday March 29, 2013 @11:56AM (#5461319)

  zpool scrub
Re:No, ZFS really doesn't need an fsck tool by thrownaway2424 on Friday March 29, 2013 @12:04PM (#5461364)
Indeed. My story from 1999.

Worked at a company that bought a ton of NetApp filers to support a webmail service. NetApp sales engineers swear on their mothers' graves that there is no such thing as fsck for wafl, that wafl always transitions from one gold-plated consistent state to the next with no possibility of metadata inconsistency. OK.

Three months later, big outage. On-site techs report the filers display "fast wack" on the front panel. Call NetApp support. What is "fast wack"? That's the fsck. Assholes!

It turned out that the filer had got corrupt somehow, and wack itself could not comprehend a filesystem with more than 2 billion files. Inode number stored in signed int32. Major, major surgery, hotpatching of filer firmware, three days of downtime, serious negative press coverage.

Bottom line: whenever anyone tells you their filesystem is guaranteed to be consistent, kick that person right in the shins.