News for hackers. Stuff that matters.
Re:Linux futex_wait() bug – update to latest patches now by btrask on Wednesday May 13, 2015 @09:47PM (#9543058)
I'm not part of the Java world at all, but I'm starting to think that Azul Systems is one of the few groups of people who know what they're doing with regard to Linux performance and the user/kernel boundary. I recently watched some talks by Cliff Click[1] that were extremely informative, and from what I understand about their proposed kernel patches, they seem like important improvements.

This report by Gil Tene (their CTO according to Wikipedia) lends more support to that theory.

[1] https://www.youtube.com/watch?v=uL2D3qzHtqY

Re:Linux futex_wait() bug – update to latest patches now by hga on Wednesday May 13, 2015 @09:55PM (#9543071)
Unfortunately, per Gin Tene as of a couple of years ago they've given up on that approach (getting kernel patches in): https://groups.google.com/d/msg/mechanical-sympathy/UnsLM6wh...

A shame, since those would allow some really sweet things for garbage collectors, which evidently isn't a goal.

Re:Linux futex_wait() bug – update to latest patches now by click170 on Wednesday May 13, 2015 @11:08PM (#9543242)
I'm skeptical of the way the author framed the discussion around his patches, I believe the majority of people out there are genuinely not evil, people don't often reject contributions that have clear benefits without reason, and I find it's often best to get said reason straight from the horses mouth.

Does anyone have a link to that discussion? I get the sense there is more to that discussion than the author let on in his brief response.

Re:Linux futex_wait() bug – update to latest patches now by Jach on Thursday May 14, 2015 @02:04AM (#9543664)
I got the sense too. The message sharing the link to the source describes it as "incomplete and extremely buggy". Was it that way when it was first proposed to the kernel group for integration? If so that seems like a reasonable reason to reject it...

I googled for "kernel mailing list managed runtime initiative" and found this: https://lwn.net/Articles/392307/ Apparently it was never even proposed on the kernel mailing list! But then apparently the code as written wasn't ever meant to be integrated into the kernel upstream but as a sort of PoC that would maybe be a better starting point than going fresh for eventual integration... But there's lots of interesting details, and as I'm reading through the comments, things are coming back, now I remember reading about this in 2010/2011...

Re:Linux futex_wait() bug – update to latest patches now by random3 on Wednesday May 13, 2015 @10:41PM (#9543180)
Also Cliff Click is doing H2O at 0xdata http://0xdata.com/team/cliff-click/
Re:Linux futex_wait() bug – update to latest patches now by osi on Thursday May 14, 2015 @08:29AM (#9544696)
They absolutely do! If you purchase their Zing JVM, it is generally because of the very low GC pauses it brings. And after that occurs, you quickly start to notice what other things are going on on your system to cause unwanted pauses in your processes.

In order to effectively sell the JVM, they need to help you understand and reduce the non-GC pauses so you can realize the full value of your investment.

(full disclosure: happy Zing customer)

Re:Linux futex_wait() bug – update to latest patches now by matheweis on Thursday May 14, 2015 @01:25AM (#9543568)
I've been trying to track down randomly latency (> 1000ms) in the network stack between the socket buffer and the client for about 3 months now... All of the stack traces showed the app was stuck in futex_wait, but since it looked identical to an idle server, I'd convinced myself epoll_wait was at fault... All the sudden I'm wondering otherwise. We're not on Haswell though; it's not clear to me if the bug would affect other processors or not - can it?
Re:Linux futex_wait() bug – update to latest patches now by paulmd on Thursday May 14, 2015 @12:06PM (#9545905)
The guy in the article has only experienced it on Haswell. However, that doesn't mean that it doesn't exist elsewhere.
Re:Linux futex_wait() bug – update to latest patches now by matheweis on Thursday May 14, 2015 @12:52PM (#9546135)
I noticed later that the actual kernel patch references arm64, so it's definitely not limited just to Haswell:

https://github.com/torvalds/linux/commit/76835b0ebf8a7fe85be...

Re:Linux futex_wait() bug – update to latest patches now by zobzu on Thursday May 14, 2015 @01:14AM (#9543542)
"We certainly haven't seen much "INSTALL PATCHES NOW" fear mongering. And we really need it, so I'm hoping this posting will start a panic."

Yeah well its a DOS triggered by pretty particular conditions. While it's bad, starting a panic for that reminds me of the boy who cried wolf story.

But then again feel free to slap a fancy site and a codeword to it, all the cool kids do it anyway :)

Re:Linux futex_wait() bug – update to latest patches now by rincebrain on Thursday May 14, 2015 @03:22AM (#9543844)
Mostly, I'm confused as to how this has only bitten people on Haswell - did pre-Haswell just enforce a MB invisibly there for some reason, or did Haswell explicitly change some semantics?

Also, an interesting note is that the commit references this deadlocking on ARM64, so I'm guessing this probably broke on non-x86 architectures in strange ways unless I'm really missing something...

Re:Linux futex_wait() bug – update to latest patches now by the8472 on Thursday May 14, 2015 @08:40AM (#9544745)
You have to remember that the absence of a barrier does not automatically causes concurrency failures in a fail-fast manner. The barriers just provide guarantees. Your code might end up working (either always or in 99.999999% of all observed operations) just by chance. So it might simply be more visible on haswell than on other systems because it behaves differently or is more aggressive about exploting non-barriered operations.

From the mailing list:

> In our case it's reproducing on 10 core haswells only which are different than 8 cores (dual vs single ring bus and more cache coherency options). It's probably a probability matter. [...]

> Pinning the JVM to a single cpu reduces the probability of occurrence drastically (from a few times a day to weeks) so I'm guessing latency distributions may have an effect.

Re:Linux futex_wait() bug – update to latest patches now by wfunction on Thursday May 14, 2015 @06:47AM (#9544307)
I'm wondering the same thing. The one thing I know that changed in Haswell is that the transactional memory instructions were found to be broken, but I assume those aren't the issue here...
Re:Linux futex_wait() bug – update to latest patches now by whoopdedo on Wednesday May 13, 2015 @08:44PM (#9542877)
The fix HAS been applied in Debian stable, as of 04 Nov 2014.
Re:Linux futex_wait() bug – update to latest patches now by nickysielicki on Thursday May 14, 2015 @01:51AM (#9543626)
Probably should prefer to the codename in this case, because debian stable is jessie as of a couple weeks ago.
Re:Linux futex_wait() bug – update to latest patches now by buster on Thursday May 14, 2015 @04:44AM (#9544023)
Which was wheezy in Nov 2014
Re:Linux futex_wait() bug – update to latest patches now by 0942v8653 on Thursday May 14, 2015 @07:39AM (#9544473)
Yes, and Jessie ships with 3.16 (and that is probably what I have). Of course, it's a very simple fix and should be easy to backport.
Re:Linux futex_wait() bug – update to latest patches now by whoopdedo on Thursday May 14, 2015 @02:12PM (#9546610)
I meant Jessie. And Wheezy never had 3.14+ so of course it wouldn't be affected. I was saying the fix was applied six months ago.
Re:Linux futex_wait() bug – update to latest patches now by kasabali on Thursday May 14, 2015 @06:24AM (#9544258)
> For some reason, people seem to not have noticed this or raised the alarm. We certainly haven't seen much "INSTALL PATCHES NOW" fear mongering. And we really need it, so I'm hoping this posting will start a panic.

Should we also alert the President? Maybe OP is only talking regarding to the ml he has posted on but we're out of context here on HN? Only affected systems in production seems to be RHEL 6.6 on Haswell.

Ubuntu 14.04/Debian 8: have the fix for a long time [0] [1]

Ubuntu 12.04/Debian 7: was never affected [3] [2]. Newer enablement stack kernels for Ubuntu has the same fix as [1].

RHEL 7: OP only talks about 6.6 so I assume either it doesn't have the regression backported to it or it already has the fix

SLES: don't know, don't care.

[0] http://kernel.ubuntu.com/git/ubuntu/linux.git/log/?showmsg=1...

[1] http://kernel.ubuntu.com/git/ubuntu/linux.git/log/?showmsg=1...

[2] https://git.kernel.org/cgit/linux/kernel/git/stable/linux-st...

[3] http://kernel.ubuntu.com/git/ubuntu/ubuntu-precise.git/tree/...

Re:Linux futex_wait() bug – update to latest patches now by buster on Thursday May 14, 2015 @06:30AM (#9544272)
Not true, it is architecture independent (commit also mentions Android) and i can find this bug backported to atleast RHEL5.11 with 2.6.18-404.

Also, i don't believe your queries are a sufficient check at all.

You can clearly find the missing default case in http://kernel.ubuntu.com/git/ubuntu/ubuntu-precise.git/tree/... . So i guess that means ubuntu precise is/was also affected?

Same in your other link https://git.kernel.org/cgit/linux/kernel/git/stable/linux-st... .

Please check your facts next time.

Re:Linux futex_wait() bug – update to latest patches now by kasabali on Thursday May 14, 2015 @06:54AM (#9544323)
Thanks, I've corrected the links, please check them again to see they are pre-regression versions [1]

> Not true, it is architecture independent (commit also mentions Android)

I saw it (and arm64 comment in this thread) but I didn't include them because I don't think it would be a production-serious issue there. Thanks for clarification.

> Please check your facts next time.

Thanks to your post I've corrected links, but no need for such an aggressive tone, huh? Maybe you should try to be more polite in your next refutal tries?

[1] https://github.com/torvalds/linux/commit/b0c29f79ecea0b6fbce...

Re:Linux futex_wait() bug – update to latest patches now by buster on Thursday May 14, 2015 @07:03AM (#9544357)
Please note that i was just looking for the missing default case. It might still be the case that the bug is not in precise..

The patch also replaces atomic_inc() with futex_get_mm():

  -		atomic_inc(&key->private.mm->mm_count);
  +		futex_get_mm(key); /* implies MB (B) */
And your links are using atomic_inc().. What that means with regards to this bug? I don't know.
Re:Linux futex_wait() bug – update to latest patches now by kasabali on Thursday May 14, 2015 @07:11AM (#9544381)
Fix commit and OP claims bug was introduced in [1], which has been never backported to 3.2.x, so I assume lack of default case was not a problem before that commit.

> And your links are using atomic_inc().. What that means with regards to this bug? I don't know.

It means they are simply old versions and they were never affected by the bug, if we believe the commit message of the fix commit and OP.

[1] https://github.com/torvalds/linux/commit/b0c29f79ecea0b6fbce...

Re:Linux futex_wait() bug – update to latest patches now by buster on Thursday May 14, 2015 @08:23AM (#9544672)
Or not recognized.. But you are right, i suppose that the atomic_inc() doesn't need a memory barrier because it's atomic and the new function may need one, guessing.
Re:Linux futex_wait() bug – update to latest patches now by snits on Thursday May 14, 2015 @01:41PM (#9546416)
Where are you seeing this backported to 5.11 or 2.6.18-404? I'm pretty sure the changeset that introduced the problem has not been backported to RHEL5 kernels.
Re:Linux futex_wait() bug – update to latest patches now by buster on Saturday May 16, 2015 @04:08AM (#9555375)
I was only looking for the missing switch case in that function (which was what has been fixed by Linus). My conclusion that this missing case is showing the bug was probably not right.
Re:Linux futex_wait() bug – update to latest patches now by planckscnst on Wednesday May 13, 2015 @09:28PM (#9543004)
My eyes jumped straight to "%$^!" and I wondered what a shell expansion had to do with a futex_wait bug. Then I briefly tried to parse it and only then read the sentence. I wondered how many others this happened to.
Re:Linux futex_wait() bug – update to latest patches now by AceJohnny2 on Wednesday May 13, 2015 @09:39PM (#9543036)
You know you've been writing too much Perl or shell scripts when...
Re:Linux futex_wait() bug – update to latest patches now by olalonde on Thursday May 14, 2015 @04:39AM (#9544013)
https://github.com/torvalds/linux/commit/76835b0ebf8a7fe85be...

30 lines commit message for 2 lines of code. Kernel developers sure are disciplined.

Re:Linux futex_wait() bug – update to latest patches now by random3 on Wednesday May 13, 2015 @09:02PM (#9542932)
I've been hunting this for weeks now, it locks down JVMs, but everything seems to indicate it's either glibc or kernel.
Re:Linux futex_wait() bug – update to latest patches now by buster on Thursday May 14, 2015 @04:50AM (#9544039)
It would be nice to know which distros are affected by this bug. Particularly, i had unexplainable JVM lockups on RHEL 5.11 this week after it was upgraded. Seeing that 6.6 and 5.11 both were released in 2014 with a 2.6.x kernel, i can imagine this bug also applying to RHEL5..

Also, does anyone know if there is some RHEL errata about this bug?

edit: I just looked at the RedHat applied patches for RHEL5.11 linux 2.6.18-398 and this bug was also introduced in the RHEL5.11 series (not sure if a subsequent kernel version fixes this)

Re:Linux futex_wait() bug – update to latest patches now by minaguib on Wednesday May 13, 2015 @10:16PM (#9543116)
Can anyone recommend a way to check a linux server for whether it's running on a Haswell CPU ?

I'm guessing perhaps checking /proc/cpuinfo for the XEON version v3, or looking for flags 'hle|rtm|tsx' would work - but something more definitive would help with mass-auditing.

Re:Linux futex_wait() bug – update to latest patches now by random3 on Wednesday May 13, 2015 @10:33PM (#9543164)
v3 in cpuinfo should do it
Re:Linux futex_wait() bug – update to latest patches now by minaguib on Thursday May 14, 2015 @01:45PM (#9546440)
Do you know if this also stands for non-XEONs ?
Re:Linux futex_wait() bug – update to latest patches now by azinman2 on Wednesday May 13, 2015 @08:08PM (#9542792)
Any word on affected Ubuntu distros and if/when they're going to patch?
Re:Linux futex_wait() bug – update to latest patches now by kasabali on Thursday May 14, 2015 @06:05AM (#9544214)
Trusty [1] and Utopic[2] already have the fix. Vivid should be unaffected since it uses 3.19 anyway.

[1] http://kernel.ubuntu.com/git/ubuntu/ubuntu-trusty.git/log/?q...

[2] http://kernel.ubuntu.com/git/ubuntu/ubuntu-utopic.git/commit...

Re:Linux futex_wait() bug – update to latest patches now by scott_karana on Wednesday May 13, 2015 @08:13PM (#9542804)
Based on the Linux kernel range of 3.14 to 3.18 inclusive, and this[1] list of Ubuntu kernel versions, I believe only 14.10 (Utopic Unicorn) would even be affected.

1 http://askubuntu.com/questions/517136/list-of-ubuntu-version...

EDIT: Turns out 14.04.02 LTS can also optionally use 3.16: http://www.omgubuntu.co.uk/2015/02/ubuntu-14-04-2-lts-releas...

Re:Linux futex_wait() bug – update to latest patches now by buster on Thursday May 14, 2015 @06:42AM (#9544296)
You can find this bug backported to many 2.6.x kernels, so don't rely on the 3.14 version number.. See my other comment regarding Ubuntu and RHEL https://news.ycombinator.com/item?id=9544272

But i didn't check if there are updated kernels for those versions in Ubuntu.. Atleast for RHEL5.11 it looks to me that the -404 kernel is the latest...

Re:Linux futex_wait() bug – update to latest patches now by yshalabi on Wednesday May 13, 2015 @11:49PM (#9543348)
This is why programmability is important. This is why being able to achieve performance of relaxed memory models with a more intuitive SC memory model should be a top objective for Intel and architecture researchers..
Re:Linux futex_wait() bug – update to latest patches now by ape4 on Wednesday May 13, 2015 @10:35PM (#9543170)
The Linux kernel has unit tests, right
Re:Linux futex_wait() bug – update to latest patches now by MichaelCrawford on Thursday May 14, 2015 @06:47AM (#9544306)
It is difficult to design meaningful unit tests for preemptively multitasking, protected memory operating system kernels. I don't know that it's actually impossible, but difficult.

While there are many advantages to unit testing, kernels are typically tested from userspace.

Some tests are laughably simple. I panicked the OS X kernel a while back with a shell script that repeatedly loaded and unloaded my kernel extension. Only a minute or two was required for the panic.

Apple fixed the panic but never told me how they screwed up.

EDIT: Of significant concern is how the kernel deals with the electrical circuitry. While the kernel is implemented in software, the reason we even have kernels, is so that end-user code doesn't have to understand much about physics.

AMCC - since acquired by LSI - sold some high-end RAID Host Bus Adapters. We had quite a significant problem with motherboard support. We had to test our cards on a whole bunch of different motherboards as well as PCI expansion chassis.

One might protest that "PCI is a standard!" but what we have is what we can buy at Microcenter. :-/

While not all of the kernel is concerned with physical hardware, much of it is. It's not really possible to write unit tests for the parts that have to deal with edge cases in electrical circuitry.

Re:Linux futex_wait() bug – update to latest patches now by caf on Wednesday May 13, 2015 @11:14PM (#9543264)
This is a race triggered by a particular reording of memory accesses as seen by different cores. It's the kind of thing that doesn't necessarily show up in a unit test anyway.
Re:Linux futex_wait() bug – update to latest patches now by exDM69 on Thursday May 14, 2015 @04:52AM (#9544042)
Exactly. This kind of bugs require stress testing to reveal. You might need to run it for minutes, hours or days to get it reproduced. And this bug might not come up with the hardware you are running.

Code dealing with memory barriers in SMP systems is non trivial to write, review and test. Everything is hardware specific, timing dependent and non-deterministic. Simple unit tests are useless for this kind of tasks, it needs stress testing on different hardware and a variety of workloads.

Re:Linux futex_wait() bug – update to latest patches now by marktangotango on Thursday May 14, 2015 @08:59AM (#9544840)
Reminds me of a passage from Tracy Kidders "Soul of a new Machine" [1]. The guys at Data General were implement one of the first pipelined 32bit processors for a mini computer in the late 70's. In the book Kidder talks about how they had a gate level simulator implemented in software that allowed them trouble shoot timing issues. Makes me wonder if a similar simulator could be useful to test and/or debug these types of issues.

Great book, highly recommend it. It won the Pulitzer.

[1] http://en.wikipedia.org/wiki/The_Soul_of_a_New_Machine

Re:Linux futex_wait() bug – update to latest patches now by fragmede on Wednesday May 13, 2015 @11:46PM (#9543336)
In case you're not trolling: not really, not officially.

Unpopular features on less common architectures are frequently broken for large stretches of time, and go unnoticed until someone complains. Open source really exemplifies the squeaky wheel getting the grease, which is kind of sad.

Places where Linux is popular undoubtedly have their own internal private test suites, especially for features less popular on bleeding edge kernels (eg S390 arch support or Infiniband)

It would be hard to get any sort of good coverage with unit tests, too, but that shouldn't be a reason to avoid trying.

Re:Linux futex_wait() bug – update to latest patches now by dasil003 on Thursday May 14, 2015 @12:10AM (#9543399)
> It would be hard to get any sort of good coverage with unit tests, too, but that shouldn't be a reason to avoid trying.

Could a large but spotty unit test suite inspire false confidence that led to be being less careful about signing off on changes and thus decrease overall quality?

Re:Linux futex_wait() bug – update to latest patches now by mburns on Thursday May 14, 2015 @03:37AM (#9543878)
Could it? Sure.

Of course, kernel devs were already confident enough to merge breaking code without the added confidence of a partial unit test suite in place.

Re:Linux futex_wait() bug – update to latest patches now by caf on Thursday May 14, 2015 @06:35AM (#9544280)
It sort of depends on the area - eg. for filesystems there is xfstests: http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/cmds/xfstests.gi...
Re:Linux futex_wait() bug – update to latest patches now by nitrogen on Thursday May 14, 2015 @02:04AM (#9543663)
There is the independent Phoronix Test Suite that is run automatically to check for performance regressions: https://en.wikipedia.org/wiki/Phoronix_Test_Suite
Re:Linux futex_wait() bug – update to latest patches now by koverstreet on Wednesday May 13, 2015 @10:52PM (#9543207)
Hah
Re:Linux futex_wait() bug – update to latest patches now by userbinator on Thursday May 14, 2015 @12:04AM (#9543386)
They mention seeing this bug appear on Haswell CPUs, and nothing about any other x86 - is it a Haswell-specific bug?
Re:Linux futex_wait() bug – update to latest patches now by matheweis on Thursday May 14, 2015 @01:48AM (#9543620)
Looks to be more than just Haswell. I was wondering this too, and just noticed this comment on the (fix) patch: "the problem (user space deadlocks) can be seen with Android bionic's mutex implementation on an arm64 multi-cluster system."
Re:Linux futex_wait() bug – update to latest patches now by mikerichards on Wednesday May 13, 2015 @11:11PM (#9543254)
from the patch I see, isn't the problem that C doesn't enforce a default case?
Re:Linux futex_wait() bug – update to latest patches now by asveikau on Thursday May 14, 2015 @01:08AM (#9543524)
Was waiting for an HN user to come forward and say that a processor specific synchronization primitive bug is C's fault... Hilarious.

Much more important what the code in that new default block is doing. It's a memory fence.

Re:Linux futex_wait() bug – update to latest patches now by cremno on Thursday May 14, 2015 @12:35AM (#9543454)
GCC has a diagnostic for that (-Wswitch-default) but see mirashii's comment.
Re:Linux futex_wait() bug – update to latest patches now by danieltillett on Wednesday May 13, 2015 @11:38PM (#9543315)
The problem is not that there is not an enforced default case, but that it was not in the coding standards. I always have a default case, but this is in my coding standards.
Re:Linux futex_wait() bug – update to latest patches now by mirashii on Wednesday May 13, 2015 @11:44PM (#9543332)
Neither of these things actually fix the problem, which is in particular the content of the default case. default: ; break; would be just as buggy.
Re:Linux futex_wait() bug – update to latest patches now by gpvos on Thursday May 14, 2015 @01:15AM (#9543546)
The commit message says the code was reviewed quite thoroughly. So I think a break-only default case where the other two cases had an /* implied MB */ comment would likely have been noticed. So in this particular case, there is a fair chance that such a warning would have helped.