I've been having flashbacks to troubleshooting some particularly thorny unreliable boot stuff several years ago. In the end tracked that one down to the fact that device order was changing somewhat randomly between commits (deterministically, though, so the same kernel from the same commit would always have devices return in the same order), and part of the early boot process was unwittingly dependent on particular network device ordering due to an annoying bug. The kernel has never made any guarantees about device ordering, so the kernel was behaving just fine.

That one was.. fun. First time I've ever managed to identify dozens of commits widely dispersed within a large range, all seem to be the "cause" of the bug, while clearly having nothing to do with anything related to it, and having commits all around them be good :)

That sounds like an interesting story! Did it result in a kernel patch or at least a blog post?

That one didn't. The bug wasn't actually in the kernel, it was in the klibc / klibc-utils.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852480 was the ultimate bug/patch that came out of it, Canonical were a vendor for us, and helped do the investigation, https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348.

The dhcp client in the klibc-utils had a bug in how it handled multiple interfaces, in that it didn't create separate sockets per interface, as it enumerated through them it would clobber the previous one. It validated the destination of the received DHCP response, and silently dropped it if it wasn't for the interface the socket was for.

The DHCP server was only listening on one of the two interfaces, and so if that interface got enumerated second, all was well and good. The socket was for it, response would be accepted. When it came up first, the clobbered socket meant the dhcp response would be ignored.

I bisected so many times and mostly just got confused. The engineer at Canonical dug in and found the actual bug.

I gave that reproducer a try and it failed after 1968 iterations.

* CPU: Intel(R) Core(TM) i9-9900KS

* qemu: qemu-kvm-7.2.1-2.fc38.x86_64

* host kernel: 6.3.6-200.fc38.x86_64

* guest kernel: 6.4.0-0.rc6.48.fc39.x86_64 (grabbed latest from mirrors.kernel.org/fedora since fedoraproject.org DNS is down and I can't access koji)

Log:

    <...>
    1966... 1967... 1968...

    [    0.075343] LSM: initializing lsm=lockdown,capability,yama,bpf,landlock,integrity
    [    0.075514] Yama: becoming mindful.
    [    0.075514] LSM support for eBPF active
    [    0.075514] landlock: Up and running.
    [    0.075514] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
    [    0.075514] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
    [    0.075514] x86/cpu: User Mode Instruction Prevention (UMIP) activated
    [    0.075514] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
    [    0.075514] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
    [    0.075514] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
    [    0.075514] Spectre V2 : Mitigation: Enhanced / Automatic IBRS
    [    0.075514] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
    [    0.075514] Spectre V2 : Spectre v2 / PBRSB-eIBRS: Retire a single CALL on VMEXIT
    [    0.075514] RETBleed: Mitigation: Enhanced IBRS
    [    0.075514] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
    [    0.075514] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl
    [    0.075514] TAA: Mitigation: TSX disabled
    [    0.075514] MMIO Stale Data: Vulnerable: Clear CPU buffers attempted, no microcode
    [    0.075514] SRBDS: Unknown: Dependent on hypervisor status
    [    0.075514] Freeing SMP alternatives memory: 48K
    *** ERROR OR HANG ***

I'll try reverting f31dcb152a3 and testing again later. Happy to test anything else if needed.

Yup, that's the bug. If it does away after reverting the commit, that would be interesting too. I don't have any other suggestions.

I tested with 6.4.0-0.rc6.48.fc39.x86_64 + f31dcb152a3 revert and all 10000 iterations succeeded (same hardware and environment as my previous post).

To guarantee that there's absolutely no other difference between the two tests, I took the source RPM, added the commit f31dcb152a3 diff + `%patch -P 2 -R`, and built the kernel RPM with mock.

I wasn't able to reproduce this with 10k iterations on arch, I'm probably doing something wrong. Does the host kernel matter?

Host kernel: 6.1.33

Guest kernel: 6.4-rc6

Guest config: http://oirase.annexia.org/tmp/config-bz2213346

QEMU: 8.0.2

Hardware: AMD Ryzen 7 3700X CPU @ 4.2GHz

> Does the host kernel matter?

Honestly I don't know! We've seen it appear with host kernel 6.2.15 (https://bugzilla.redhat.com/show_bug.cgi?id=2213346#c5) but I'm not aware of anyone either reproducing or not reproducing it with earlier host kernels. All your other config looks right.

vmlinuz-6.4.0-0.rc6.48.fc39.x86_64 failed on my 6.0.0 host after 249 iterations.

We had another report that it happens on RHEL 8 host, which is a very much older (franken) kernel.

I noticed it hangs in similar way when you insert msleep anywhere before smp_prepare_cpus in kernel_init_freeable. But I have no idea whether sleeping is valid here.

Can confirm - just adding msleep(1) there causes the same failure. I'm also unclear if sleeping here is valid or not.

Host kernel: 6.1.33-1-lts (Arch) Guest kernel: 6.4-rc6 defconfig QEMU: 8.0.2 (Arch) CPU: AMD EPYC 74F3

1242 iterations :

  [    0.015088] printk: console [ttyS0] enabled
  [    0.055882] ACPI: Core revision 20230331
  [    0.056124] APIC: Switch to symmetric I/O mode setup
  [    0.056867] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2e204823bcd,   max_idle_ns: 440795224253 ns
  [    0.057467] Calibrating delay loop (skipped) preset value.. 6399.99 BogoMIPS (lpj=3199998)
  [    0.057924] pid_max: default: 32768 minimum: 301
  [    0.058194] LSM: initializing lsm=capability,integrity
  [    0.058464] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
  [    0.058464] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
  [    0.058464] x86/cpu: User Mode Instruction Prevention (UMIP) activated
  [    0.058464] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127
  [    0.058464] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0
  [    0.058464] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
  [    0.058464] Spectre V2 : Mitigation: Retpolines
  [    0.058464] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
  [    0.058464] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT
  [    0.058464] Spectre V2 : Enabling Restricted Speculation for firmware calls
  [    0.058464] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
  [    0.058464] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl
  [    0.058464] Freeing SMP alternatives memory: 48K
  *** ERROR OR HANG ***

After reverting commit f31dcb152a3d0816e2f1deab4e64572336da197d :

40000 iterations (4 runs) = "test ok"

Looks like you have a trigger, but no root cause (yet). Doesn't matter anyway...revert and work it out later. The root cause bug is still in there somewhere, waiting to be triggered another way...

It's the Linux kernel so I'm in no position to revert the patch.