News for hackers. Stuff that matters.
Re:Linux: Ext4 data corruption in 6.1.64-1 by GauntletWizard on Sunday December 10, 2023 @01:29AM (#38589603)
As someone who is heavily dependent on zfs, this is a little bit of reassurance. Sad that data corruption exists in any file system that is shipped too many people, but reassuring that it happens to the most stable and least "interesting" in the newness sense file systems.

ZFS's own recent file system corruption issue is in roughly the same category of edge case, but accessible to reasonable if niche workloads.

Re:Linux: Ext4 data corruption in 6.1.64-1 by Lt_Riza_Hawkeye on Sunday December 10, 2023 @02:38AM (#38589858)
The problem here was Debian's distribution process. Any distro compiling linux from the releases on kernel.org was not affected.
Re:Linux: Ext4 data corruption in 6.1.64-1 by peppermint_gum on Sunday December 10, 2023 @03:22AM (#38590048)
This is not accurate. As you can see in the changelogs, the problematic commit made it to 6.1.64 and the fix was merged in 6.1.66:

"properly sync file size update after O_SYNC direct IO": https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.6...

"update ki_pos a little later in iomap_dio_complete": https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.6...

This post explains the relationship between the two commits: https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuh...

Re:Linux: Ext4 data corruption in 6.1.64-1 by Roark66 on Sunday December 10, 2023 @04:30AM (#38590344)
What if you build your own kernel from kernel.org, but you use Debian tools to make dpkgs? (and you use old config)?
Re:Linux: Ext4 data corruption in 6.1.64-1 by burtness on Sunday December 10, 2023 @03:22AM (#38590047)
This is a misreading of the bug. It is from upstream stable kernels before 6.5 that include commit 91562895f803 but not 936e114a245b6[1].

In this case Debian's current process is good - it's kernels track kernel.org stable releases. This debian bug is responsibly flagging "for visibility" that a serious bug has been discussed and fixed upstream.

[1] https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuh...

Re:Linux: Ext4 data corruption in 6.1.64-1 by csdvrx on Sunday December 10, 2023 @06:32AM (#38590851)
Do you have a link to just the 2 patches (in patch form) that could be before on a kernel tarball to revert these 2 changes?
Re:Linux: Ext4 data corruption in 6.1.64-1 by uselpa on Sunday December 10, 2023 @03:07AM (#38589985)
Are you sure about that (genuine question)? The linked discussion involves a Suse engineer and a request to the kernel maintainer directly, not to a Debian packager-
Re:Linux: Ext4 data corruption in 6.1.64-1 by burtness on Sunday December 10, 2023 @03:22AM (#38590050)
No, its an upstream bug being discussed in the debian bug tracker
Re:Linux: Ext4 data corruption in 6.1.64-1 by Blikkentrekker on Sunday December 10, 2023 @02:50AM (#38589912)
So this is another bug introduced by Debian itself by patching things?

I remember that there was a fairly severe one which was caused by patching OpenSSL I think? But I remember the change they made being fairly weird and no one understood why but it was easy to see that it would introduce a vulnerability.

Re:Linux: Ext4 data corruption in 6.1.64-1 by mordae on Sunday December 10, 2023 @09:09AM (#38591593)
No, the issue was caused by backporting a patch A but not backporting a patch B. Sadly in this case the overall behavior after applying just A was broken when issuing direct IO writes.
Re:Linux: Ext4 data corruption in 6.1.64-1 by keep_reading on Sunday December 10, 2023 @10:14AM (#38592033)
I remember when they made 2 CVEs back to back in Apache by incorrect back porting of changes
Re:Linux: Ext4 data corruption in 6.1.64-1 by remram on Sunday December 10, 2023 @11:27AM (#38592657)
No, it exists in the upstream release.
Re:Linux: Ext4 data corruption in 6.1.64-1 by upon_drumhead on Sunday December 10, 2023 @01:43AM (#38589650)
Yeah, all the threads full of folks bashing on ZFS because their trusty ext won’t corrupt data should be eating crow right about now, but I’m sure they’ll hop in and explain why this bug is different.
Re:Linux: Ext4 data corruption in 6.1.64-1 by uxp8u61q on Sunday December 10, 2023 @02:01AM (#38589714)
This isn't a sports tournament, nobody is cheering or bashing "teams". If ZFS has bugs and caveats, they should be called out. Same for ext4.
Re:Linux: Ext4 data corruption in 6.1.64-1 by GauntletWizard on Sunday December 10, 2023 @02:44AM (#38589884)
Yes, it really is. People have irrational attachments and team-lime cheering for whatever they feel attached to. When ZFS has issues, you see it's detractors out in mass. Whenever there's a issue in a C coreutil, there's a ton of rusticles out to complain that about writing in a "memory unsafe" language. When there's a compiler bug in Rust of the same severity, you can expect down votes. When a core rust developer is exposed as a psychotic pedophile, there's crickets - it's not news because it happens every month.

I wouldn't even bother to mention it if it weren't for the 15-ish votes on my comment, a total score of 0

Re:Linux: Ext4 data corruption in 6.1.64-1 by baz00 on Sunday December 10, 2023 @03:03AM (#38589971)
Well said. Twisted partisanism.

I’m the middle ground. It all sucks in one way or another but we’re generally trying to our best.

Re:Linux: Ext4 data corruption in 6.1.64-1 by reanimus on Sunday December 10, 2023 @06:53AM (#38590939)
> When a core rust developer is exposed as a psychotic pedophile, there's crickets - it's not news because it happens every month.

bro what on earth are you on about

Re:Linux: Ext4 data corruption in 6.1.64-1 by megous on Sunday December 10, 2023 @01:51AM (#38589678)
It's not an extfs bug, it's a "stable kernel" process bug. Linux master branch/mainline was never affected by this bug. Someone cherry-picked a wrong patch into stable without realizing the consequences, and it was caught too late.

Stable process is cherry pincking thousands to tens of thousands of patches from the current master kernel branch into years old kernel branches, spraying tens of thousands of emails at original patch authors, hoping (With some limited testing on top) that the resulting frankenkernels will still work and all these patches will have satisfied dependencies applied, too.

I don't trust this process very much, and just run the latest stable branch for the latest kernel release, only. Staying with older stable release branch for too long seems too risky, unless you're some bigcorp that can afford the testing required, or you're running some highly mainstream setup that is probably covered by tests done by the stable team, and testing teams they cooperate with.

Re:Linux: Ext4 data corruption in 6.1.64-1 by uselpa on Sunday December 10, 2023 @03:11AM (#38590004)
I understand your reasoning, but running a popular « stable » distro in a popular configuration will help detect bugs much faster. I’m not sure this would be the case for the very latest kernel.
Re:Linux: Ext4 data corruption in 6.1.64-1 by megous on Sunday December 10, 2023 @05:17AM (#38590572)
A bunch of popular distros run the latest kernel.
Re:Linux: Ext4 data corruption in 6.1.64-1 by keep_reading on Sunday December 10, 2023 @10:16AM (#38592044)
Nobody ships vanilla kernels. Sure the reference implementation was unaffected but users never experience vanilla Linux kernels
Re:Linux: Ext4 data corruption in 6.1.64-1 by dezgeg on Sunday December 10, 2023 @11:04AM (#38592441)
Distros like Arch and NixOS use the vanilla kernel with very small amount (single digits) of patches on top. For the purposes of ext4, it's very likely to be as-is with vanilla. (I am counting the official linux-stable releases from kernel.org as 'vanilla' here.)
Re:Linux: Ext4 data corruption in 6.1.64-1 by megous on Sunday December 10, 2023 @11:52AM (#38592879)
Fedora, openSUSE tumbleweed, Manjaro,...
Re:Linux: Ext4 data corruption in 6.1.64-1 by keep_reading on Sunday December 10, 2023 @04:30PM (#38595250)
> Fedora

You know it's not that hard to debunk this:

> You'll then be left with a kernel-6.X.? directory, containing both an unpatched 'vanilla-6.X.?' dir, and a linux-6.X.?-noarch hardlinked dir which has the Fedora patches applied. [1]

WOW fedora patches! Sounds to me like they're not shipping vanilla.

[1] https://fedoraproject.org/wiki/Kernel

Re:Linux: Ext4 data corruption in 6.1.64-1 by megous on Monday December 11, 2023 @07:19AM (#38599952)
That's not what my original argument was about. So not sure what you're debunking.

I have problem with backaptching 10s of thousands of changes to years old kernels by people who don't really understand the changes or consequences, like 5.4, 5.15, 6.1 or whatever. Not patching up 6.6 or 6.7 kernel with a few out of tree patches, where they mostly understand what they're doing and can test the limited set of changes they're applying.

Re:Linux: Ext4 data corruption in 6.1.64-1 by keep_reading on Wednesday December 13, 2023 @02:29PM (#38632923)
If they change anything it's no longer vanilla. Anything at all. No matter how insignificant.
Re:Linux: Ext4 data corruption in 6.1.64-1 by keep_reading on Sunday December 10, 2023 @04:33PM (#38595267)
Even a single patch to something like default kernel config parameters means it's not vanilla. I'm not arguing that there are significant patches, only that nobody seems to ship a true vanilla kernel.
Re:Linux: Ext4 data corruption in 6.1.64-1 by jcalvinowens on Sunday December 10, 2023 @09:29PM (#38597014)
> Even a single patch to something like default kernel config parameters means it's not vanilla.

Anybody who cares that much is very likely already compiling their own kernels (I speak for myself here). It doesn't make sense for distros to do the extra work to support it.

Just expanding on this a bit: my Debian laptop has a Kconfig with modules disabled that only includes the exact set of drivers it needs. It takes ten minutes to rebuild on the laptop when I pull from git. Even if Debian did all the work to let me automatically install the latest vanilla kernel... I'd still build it myself.

Re:Linux: Ext4 data corruption in 6.1.64-1 by 404mm on Sunday December 10, 2023 @10:56AM (#38592380)
You’re forgetting the btrfs folks.

— someone who lost data due to btrfs bug

Re:Linux: Ext4 data corruption in 6.1.64-1 by nolist_policy on Sunday December 10, 2023 @03:06PM (#38594519)
Note though that unlike zfs, ext4 has a battle-tested fsck.

Combine ext4's dumb but robust approach to journaling and robust metadata layout (for example inodes are statically allocated) with fsck.ext4 (which got refined for years) and you can recover from any situation.

To give you an example, fsck.ext4 will happily carve a working filesystem out of random data, as long as there is a valid superblock. Seriously - try it yourself:

  # create a working filesystem image
  dd if=/dev/zero of=test1.img bs=1M count=256
  /sbin/mkfs.ext4 test1.img
  # write file with random data
  dd if=/dev/urandom of=test2.img bs=1M count=256
  # copy superblock into random file
  dd if=test1.img of=test2.img bs=1024 count=4 seek=1 skip=1 conv=notrunc
  /sbin/fsck.ext4 -fy test2.img
  sudo mount test2.img /mnt/somewhere
Re:Linux: Ext4 data corruption in 6.1.64-1 by mustache_kimono on Sunday December 10, 2023 @04:06PM (#38595062)
I'm trying to understand what exactly this is testing/demonstrating.

Like: "Wow, but why should I care?" I'm not sure being able to fsck a random disk image shows resiliency.

Re:Linux: Ext4 data corruption in 6.1.64-1 by grrandalf on Monday December 11, 2023 @05:05PM (#38606144)
(not the author) The metadata is a contiguous range of disk blocks. I think the intuition is that such layouts are likely to require simpler filesystem code to manipulate. [versus (iirc) ZFS which may have metadata blocks scattered throughout the disk, probably requiring more intricate code to keep track of].
Re:Linux: Ext4 data corruption in 6.1.64-1 by mustache_kimono on Tuesday December 12, 2023 @03:55PM (#38618376)
> [versus (iirc) ZFS which may have metadata blocks scattered throughout the disk, probably requiring more intricate code to keep track of].

So -- you think redundant metadata is a bad thing? Try wiping your metadata and then trying to fsck that random disk image.

Again, has this been the source of data corruption you're aware of? This seems like a "Maybe, it could be this way, but I don't know" kind of take.

Re:Linux: Ext4 data corruption in 6.1.64-1 by 1000thVisitor on Sunday December 10, 2023 @05:12PM (#38595607)
> To give you an example, fsck.ext4 will happily carve a working filesystem out of random data

What application does this have for recovering user data? Would it not be more interesting to see how much of the user data can be recovered? Or are you implying that the random data can symbolize the user data here and much of it would be recovered?