I booted Linux 292,612 times

Re:I booted Linux 292,612 times by Twirrim on Wednesday June 14, 2023 @03:28PM (#36331275)

I've been having flashbacks to troubleshooting some particularly thorny unreliable boot stuff several years ago. In the end tracked that one down to the fact that device order was changing somewhat randomly between commits (deterministically, though, so the same kernel from the same commit would always have devices return in the same order), and part of the early boot process was unwittingly dependent on particular network device ordering due to an annoying bug. The kernel has never made any guarantees about device ordering, so the kernel was behaving just fine.

That one was.. fun. First time I've ever managed to identify dozens of commits widely dispersed within a large range, all seem to be the "cause" of the bug, while clearly having nothing to do with anything related to it, and having commits all around them be good :)

[ Parent | on HN ]

Re:I booted Linux 292,612 times by pcthrowaway on Wednesday June 14, 2023 @11:21PM (#36336025)

That sounds like an interesting story! Did it result in a kernel patch or at least a blog post?

[ Parent | on HN ]

Re:I booted Linux 292,612 times by Twirrim on Thursday June 15, 2023 @08:46PM (#36349321)

That one didn't. The bug wasn't actually in the kernel, it was in the klibc / klibc-utils.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852480 was the ultimate bug/patch that came out of it, Canonical were a vendor for us, and helped do the investigation, https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348.

The dhcp client in the klibc-utils had a bug in how it handled multiple interfaces, in that it didn't create separate sockets per interface, as it enumerated through them it would clobber the previous one. It validated the destination of the received DHCP response, and silently dropped it if it wasn't for the interface the socket was for.

The DHCP server was only listening on one of the two interfaces, and so if that interface got enumerated second, all was well and good. The socket was for it, response would be accepted. When it came up first, the clobbered socket meant the dhcp response would be ignored.

I bisected so many times and mostly just got confused. The engineer at Canonical dug in and found the actual bug.

[ Parent | on HN ]