News for hackers. Stuff that matters.
Re:Tell HN: ZFS silent data corruption bugfix – my research ... by worthless-trash on Thursday December 07, 2023 @03:40AM (#38554232)
No, you're just hearing about ZFS corruption like you've heard corruption in other filesystems, this wont be the first and last.

Its easy to feel like its being targeted with some kind of campaign, however the truth is rarely that exciting. The software is being used by more people, which means it will expose more bugs.

No software is perfect, it just now starting to be abused enough to be important enough to be talked about. This stage is entirely normal.

Re:Tell HN: ZFS silent data corruption bugfix – my research ... by buildbot on Thursday December 07, 2023 @04:12AM (#38554441)
But it's totally different than corruption in other filesystems. People are acting like upgrading to ZFS 2.2.0 ate all their data like XFS used to back in the bad old days. I remember once the power went out at my house, and the entire XFS filesystem was irreparably damaged.

This bug is super hard to trigger, and has roots back to 2006 and only surfaced with a recent coreutils update. Yet there are numerous posts in the past few days verging on hysteria. I myself was very worried until I saw posts like the grand parent comment or the linked GitHub issues.

I don't believe there's actually a campaign of ZFS haters sitting around in a storeroom somewhere posting to phoronix and HN, but it does feel like a lot of strongly held opinions are being unleashed now that ZFS had this bug.

I'm pretty sure ZFS has been abused a lot - ten years ago I was helping test petabyte scale ZFS clusters, doing tons of concurrent IO and such. I'm pretty sure Sun and Oracle tested the crap out of it.

Re:Tell HN: ZFS silent data corruption bugfix – my research ... by worthless-trash on Thursday December 07, 2023 @04:27AM (#38554531)
> But it's totally different than corruption in other

> filesystems. People are acting like upgrading to ZFS 2.2.0

> ate all their data like XFS used to back in the bad old

> days. I remember once the power went out at my house, and

> the entire XFS filesystem was irreparably damaged.

Different bugs manifest differently due to the coding, this is just the same.

ZFS wouldn't even have a hundredth of the testing of real-world use ext3/ext4 in hours. The large scale deployments of ext3/ext4 dwarf the usage of ZFS.

Re:Tell HN: ZFS silent data corruption bugfix – my research ... by lproven on Thursday December 07, 2023 @06:18AM (#38555185)
ext4 released: 2008

https://archive.ph/20120529150649/http://git.kernel.org/?p=l...

ZFS released: 2005

https://web.archive.org/web/20130619165135/https://blogs.ora...

This is Linux local bias. There is more to Unix than Linux.

Back around 2001 when work on ZFS began, I very strongly suspect that there were several orders of magnitude more storage on ZFS on Solaris in datacentres around the world than there were on all Linux filesystems combined.

Just because you are more familiar with the FOSS tool doesn't mean that it's more mature or more widely used or more deployed in production.

Re:Tell HN: ZFS silent data corruption bugfix – my research ... by worthless-trash on Friday December 08, 2023 @07:08AM (#38568020)
I would -very- strongly imagine it the other way. I guess we'll never know.
Re:Tell HN: ZFS silent data corruption bugfix – my research ... by buildbot on Friday December 08, 2023 @11:48AM (#38571067)
I worked at a company that shipped petabytes of ZFS storage for years and years. Think 42u racks with 60 drive JBODs. Hundreds of these. Hundreds. And they were a very small company. Can you imagine the scale Sun and Oracle shipped hardware at? Was there any linux based hardware company operating at the same scale?

I don’t think we had any customers request ext4 bulk data storage. OS drives sure. Everything else was ZFS. Or weirder proprietary filesystems!

Re:Tell HN: ZFS silent data corruption bugfix – my research ... by worthless-trash on Sunday December 10, 2023 @01:40AM (#38589640)
Yeah, that GPFS was real pita, i'm not talking about size comparison issues. I'm talking about runtime in the field.

You simply can't begin to understand the amount of shitty iot and android devices that exist.

Re:Tell HN: ZFS silent data corruption bugfix – my research ... by lproven on Saturday December 09, 2023 @06:35AM (#38581038)
See the earlier comment to you.

I think you have this 100% backwards. I heard similar comments to yours inside some large Linux vendors, too, and I think they betray a profound misunderstanding about the nature of large-scale Unix deployments over the years.

FOSS Unix is recent and mostly small-scale and transient deployments of containers, not long-term storage... but before that, there were decades of vast scale commercial Unix systems, which dwarf the FOSS stuff.

Re:Tell HN: ZFS silent data corruption bugfix – my research ... by worthless-trash on Sunday December 10, 2023 @01:40AM (#38589641)
How many zfs systems you think are running right now, all hours combined compared to linux ext filesystems ?
Re:Tell HN: ZFS silent data corruption bugfix – my research ... by lproven on Sunday December 10, 2023 @04:01PM (#38595003)
"Bananas are better than passion fruit because there are more bananas being grown!"

ZFS is an enterprise storage-management tool. It's not just another hard disk format, like ext4.

Not many people are running Linux boxes with root on ZFS because few distros support that.

Single-disk setups are not its strength so the number of single-disk setups of ZFS compared to ext4, which only supports single-disk setups, is irrelevant.

Even saying that, though...

I bet all FreeBSD deployments are on ZFS, and it's got a solid enterprise presence. Anything streamed off Netflix anywhere is being served by FreeBSD.

Oracle still supports Solaris. All Solaris, OpenSolaris, Illumos, Nexenta, OmniOPS, SmartOS, Tribblix... all on ZFS.

And it's still not its core area.

If you want to do a fair comparison let's hear about companies doing petascale to exascale arrays with double-digit to quadruple-digit numbers of drives.

How many of them do you reckon use ext4?

Re:Tell HN: ZFS silent data corruption bugfix – my research ... by worthless-trash on Monday December 11, 2023 @10:48PM (#38609180)
I'm not the one who moved the goal posts here though was I ?

Here, let me quote myself.

> The software is being used by more people, which means it will expose more bugs.

Re:Tell HN: ZFS silent data corruption bugfix – my research ... by kaba0 on Thursday December 07, 2023 @12:29PM (#38559187)
ZFS and ext4 is so completely different on a fundamental level that there is zero reason comparing them, it just shows your "expertise" on the topic..
Re:Tell HN: ZFS silent data corruption bugfix – my research ... by worthless-trash on Friday December 08, 2023 @07:07AM (#38568009)
I dont think it takes expertise to do basic math. But aint no value in ego measuring online.
Re:Tell HN: ZFS silent data corruption bugfix – my research ... by deepbluev7 on Thursday December 07, 2023 @09:57PM (#38564902)
To me it does feel like this has been on HN a bit more than other filesystem corruption bugs. This bug can basically only be triggered by using lseek to search for a hole. Ext4 has a very similar bug, but that requires enabling inline_data (and possibly a non-default blocksize, but that doesn't always seem to be the case): https://forums.gentoo.org/viewtopic-t-1166006.html

I don't think I ever heard about this, apart from in the context of the ZFS bug. And although inline_data is niche, ext4 as a whole I would argue is not.

Actually lseek seems to have been broken on most filesystems at some point: https://bugs.gentoo.org/891125 https://github.com/gluster/glusterfs/issues/894

And apparently apart from modern coreutils using that, it is mostly gentoo users hitting the bugs in lseek.