News for hackers. Stuff that matters.
[dead]
Re:Raid-Z Expansion Feature for ZFS Goes Live by vvatermelone on Tuesday February 08, 2022 @06:07PM (#30266167)
They "lost everything" because they failed to set up even the most basic and critical maintenance functions built into ZFS. I don't think the responsibility in this incident could fall any more squarely on the shoulders of the people who set it up.
Re:Raid-Z Expansion Feature for ZFS Goes Live by toast0 on Tuesday February 08, 2022 @11:58PM (#30269016)
I think there's a case to be made that a NAS oriented OS/distro should probably default to having a scheduled scrub, not wait for you to set one up. I don't follow LTT, but a quick look around says he was using TruNAS for at least one of the builds?

You do need a notification feature as well, which is tricky because a distro can't assume you'll setup mail properly, but maybe something like make samba go read only when the zpool is degraded could work.

Re:Raid-Z Expansion Feature for ZFS Goes Live by wolrah on Wednesday February 09, 2022 @01:05AM (#30269482)
They mentioned in their video about it how TrueNAS does do this correctly but they had manually set up this particular system on a normal distro.
Re:Raid-Z Expansion Feature for ZFS Goes Live by ksec on Wednesday February 09, 2022 @05:41AM (#30271169)
It wouldn't have happened and they would have all the error report for months if they have it on by default. Which they somehow didn't.

It's funny thinking back now when LTT asked why MKBHD didn't ask for help when he wanted a DAS.

Re:Raid-Z Expansion Feature for ZFS Goes Live by 7steps2much on Tuesday February 08, 2022 @05:50PM (#30265981)
To be fair, they didn't really understand how ZFS works and failed to set up bitrot detection.

A "clean" setup would include those, as well as either a messaging system or a regular checkup on how your FS is doing.

Re:Raid-Z Expansion Feature for ZFS Goes Live by liuliu on Tuesday February 08, 2022 @07:04PM (#30266656)
Not only that. A vdev with 15 disks each around 16 (or 18?) TiB with RAID-Z2 just asking for troubles (https://www.zdnet.com/article/why-raid-6-stops-working-in-20...). Even with 1% disk failures annually (based on Backblaze, for better disks), we are looking at least two disks fail at the same time with 15 disks around 1% chance. With 10 vdevs, that is 10% chance of any of these have failures.

They should really move to RAID-Z3 or 8 disks group. With RAID-Z3, we are looking at at least 3 disk fail at the same time with 15 disks around 0.04% chance. With 8 disks, we are looking at 0.2% chance.

Re:Raid-Z Expansion Feature for ZFS Goes Live by kalleboo on Tuesday February 08, 2022 @08:28PM (#30267451)
I've read this article and others like it, and what I don't get is if the URE rates are really that high, why I have I never seen an URE in the bi-weekly scrubs of my pool (184 TB of raw disk), aside from when a single disk was literally going bad?
Re:Raid-Z Expansion Feature for ZFS Goes Live by sliken on Tuesday February 08, 2022 @10:19PM (#30268342)
There's many contributing factors. Power supplies degrade over time, ability to maintain the correct voltages can be impaired for worst case scenarios, like running all disks flat out. Drives (and even more so older drives) can be vibration sensitive, especially in worse case scenarios involving all drives running flat out. With modern manufacturing tolerances being so low, whatever triggered the first disk in your pool to die is likely to get a second in a fairly small window.

So sure, the failures are not equally distributed and you've done well with your 184TB pool. Are you tracking the device errors, or only those that are visible to the OS? But multiple disk failures are not particularly uncommon. My experience in this space was two 16 disk servers that I set up 6 5 disk RAID5s (with one global spare per server). Within one month I had 11 of 32 disks die, and barely managed not to lose any user files, and this was not during the 1st month in production.

Scary. I've since moved to pairs of servers cross connected to pairs of 60 disk chassis (16 x 12 gbit connections per chassis) with ten 11-disk RAIDz3, 10 global spares, and 6x3.2TB of NVMe cache per server.

Re:Raid-Z Expansion Feature for ZFS Goes Live by kalleboo on Wednesday February 09, 2022 @08:22AM (#30272293)
Oh there are all kinds of reasons drives can cause errors, and you have the bathtub curve. So there's lots to take into account when designing your pool.

But the article is using the spec sheet URE rate which I'd assume looks only at the drive and doesn't take into account problems with the computer around the drive or the EOL time after the drive warranty has expired, I'd assume it was the "baseline" error rate.

> Are you tracking the device errors, or only those that are visible to the OS?

If we're talking URE like the article, that's data-loss on a disk, and the OS would always figure it out, since it would cause a ZFS checksum failure on scrub.

In this case it's not my data and not my money, so my preference is 6-drive RAIDZ2 vdevs. We've only had one disk with errors (and that one was migrated from a PC where Windows never reported any errors... of course...). The oldest 2 disks (3.5 years power-on time) have single-digit reallocated sectors in SMART so those are on course to be replaced.

I'm just curious since the argument in the article doesn't add up in my eyes.

> Within one month I had 11 of 32 disks die, and barely managed not to lose any user files, and this was not during the 1st month in production

Wow, that is some terrible luck!

Re:Raid-Z Expansion Feature for ZFS Goes Live by mustache_kimono on Tuesday February 08, 2022 @07:09PM (#30266706)
They even admit this in the video -- "No one is to blame here except us" which means "Holy shit! We really messed this up."

Yes, they should have been scrubbing their pools, but I don't think this was bit rot. Millions of data errors is not what bit rot looks like. This looks exactly like bad hardware.

Re:Raid-Z Expansion Feature for ZFS Goes Live by ksec on Wednesday February 09, 2022 @05:46AM (#30271199)
ServetheHome actually reported the hard drive LTT are using had higher failure rate than others. Even from a consumer hardware perspective. And they were suppose to be enterprise drive.

It is just bad hardware, bad setup and bad everything all cramped together.

Re:Raid-Z Expansion Feature for ZFS Goes Live by mustache_kimono on Wednesday February 09, 2022 @09:44AM (#30273158)
I've never seen a HDD fail like that, but I'm willing to believe it's possible. I have seen read/write errors because of a buggy implementation of ALPM.

But, agreed, whatever it was, if they were paying attention, they could have diagnosed and remediated well before they had any data loss.

Re:Raid-Z Expansion Feature for ZFS Goes Live by Youden on Tuesday February 08, 2022 @05:54PM (#30266031)
You can say this about literally any storage system. Unrecoverable failures can always happen, that's why you keep backups.

ZFS redundancy features aren't there to eliminate the need for backups, they're there to reduce the chance of downtime.

Re:Raid-Z Expansion Feature for ZFS Goes Live by mustache_kimono on Tuesday February 08, 2022 @06:03PM (#30266122)
This is just silly. As other have asked, how is this different from any other storage solution? (It's not.)

The only thing to glean from this video is that LTT didn't know what they were doing.

Re:Raid-Z Expansion Feature for ZFS Goes Live by namibj on Tuesday February 08, 2022 @06:29PM (#30266383)
Btrfs lets you keep metadata at a stronger bitrot-protection level fairly easily, and as long as that is still ok, you'll only use the data blocks that rotted, so at LTT's scale just a few seconds of footage in total.
Re:Raid-Z Expansion Feature for ZFS Goes Live by NavinF on Tuesday February 08, 2022 @06:34PM (#30266423)
Even if that’s true, btrfs would be entirely impractical for them because btrfs-raid6 is broken. Their petabyte server would become a 1/3 petabyte server without raidz2.
Re:Raid-Z Expansion Feature for ZFS Goes Live by namibj on Tuesday February 08, 2022 @06:50PM (#30266559)
But that's more because no one dared to try and actually fix Btrfs parity raid. Misaligned incentives causing lack of funding for something as flexible and generally resilient as Btrfs.
Re:Raid-Z Expansion Feature for ZFS Goes Live by guipsp on Tuesday February 08, 2022 @07:09PM (#30266701)
My data doesn't really care if the reason for brtfs losing data is lack of funding
Re:Raid-Z Expansion Feature for ZFS Goes Live by matheusmoreira on Tuesday February 08, 2022 @10:58PM (#30268614)
Can you elaborate on this? I see NAS companies like Synology using btrfs in their products. I don't understand why btrfs RAID 5/6 is still not stable.
Re:Raid-Z Expansion Feature for ZFS Goes Live by genghizkhan on Tuesday February 08, 2022 @11:57PM (#30269011)
Synology deploys btrfs on top of md-raid. They do not trust btrfs's raid layer.
Re:Raid-Z Expansion Feature for ZFS Goes Live by namibj on Wednesday February 09, 2022 @12:35AM (#30269267)
Because there are some write holes around iirc power failures and, more importantly, restore basically doesn't work (the code just doesn't exist in a working variant).

This isn't easy code to write, and it's Linux-Kernel-C, far from "comfy" code to write. I assume it's that kind of effect that continues to keep all competent people from fixing Btrfs's RAID 5/6 code. (If I could gather the motivation, I could probably do it, but that's far from fun to do and I'm already dire for motivation-to-code... it needs to be someone who likes that work, ideally sponsored so they don't loose money from working on that project.)

Re:Raid-Z Expansion Feature for ZFS Goes Live by mustache_kimono on Tuesday February 08, 2022 @06:48PM (#30266535)
> Btrfs lets you keep metadata at a stronger bitrot-protection level

I really have no idea what BTRFS feature this refers to, but ZFS stores redundant copies of metadata. See: redundant_metadata, https://zfsonlinux.org/manpages/0.8.0/man8/zfs.8.html

Re:Raid-Z Expansion Feature for ZFS Goes Live by namibj on Tuesday February 08, 2022 @06:52PM (#30266577)
Block group specific profiles. The normal these days is SINGLE for data and DUP for metadata.

See how one converts here: https://btrfs.readthedocs.io/en/latest/btrfs-balance.html

Re:Raid-Z Expansion Feature for ZFS Goes Live by mustache_kimono on Tuesday February 08, 2022 @07:03PM (#30266652)
Okay, I still don't understand your point.

BTRFS stores duplicate copies of the metadata. Good for BTRFS. I don't understand how this would have prevented what LTT experienced.

First, as someone else said -- no one would use a BTRFS raid5/6 array in production. No one. Second, I don't think what LTT experienced was bit rot. There were far, far too many data errors for that be simple bit rot. My guess is bad cable, bad HBA which maybe doesn't support something like ASPM, etc. Third, are we even sure the problem was corrupted metadata? Fourth, ZFS keeps redundant copies of metadata too (as do many filesystems.)

So the question is what's your point?

Re:Raid-Z Expansion Feature for ZFS Goes Live by InvaderFizz on Tuesday February 08, 2022 @11:37PM (#30268885)
Plenty of people do, like my predecessors at a former job. I then had to clean up the blast from multiple BTRFS RAID6 arrays.

Oh, you meant no one with a shred of responsibility, which I agree with.

Re:Raid-Z Expansion Feature for ZFS Goes Live by namibj on Tuesday February 08, 2022 @07:09PM (#30266703)
That you could have dialed up the redundancy for metadata to 3x or even 4x copies being stored, and just because a block got bitrot doesn't mean other blocks fail to read.

I know no one would use Btrfs raid5/6 in production, but that's because no one wanted to enough to get the required development done.

If it was just file blocks that got corrupted, there would be just very minor damage in total and a scrub that logs all damaged blocks typically proceeds at sequential read speeds from what I've observed in the past.

Re:Raid-Z Expansion Feature for ZFS Goes Live by chungy on Tuesday February 08, 2022 @07:29PM (#30266916)
ZFS already stores three logical copies of all metadata, regardless of copies parameter or vdev redundancy. The failures LTT experienced went way beyond what ZFS, or btrfs, could repair.

They made severe errors in judgment when setting up the system and failing to monitor and administer it. No system will ever protect against this. Not ZFS. Not btrfs.

Re:Raid-Z Expansion Feature for ZFS Goes Live by derkades on Tuesday February 08, 2022 @06:32PM (#30266403)
That's like saying "having 4 parity disks is dangerous, you'll lose all your data if 5 disks die!"

They were supposed to run scrubs and replace disks when they die, not let it sit for 5 years.

Re:Raid-Z Expansion Feature for ZFS Goes Live by ikiris on Tuesday February 08, 2022 @06:00PM (#30266090)
When you do things wrong, things go wrong. Film at 11.