News for hackers. Stuff that matters.
Re:Intel SA-00145: Lazy FP State Restore by kentonv on Wednesday June 13, 2018 @06:37PM (#17307589)
Andy Lutomirski noted on another thread that he unintentionally fixed this two years ago in Linux:

https://news.ycombinator.com/item?id=17304947

(He switched it to eager FP because it's faster on modern hardware.)

But a lot of people running old LTS kernels may be affected.

EDIT: Looks like Luto's change landed in kernel version 4.6. https://kernelnewbies.org/Linux_4.6#List_of_merges https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Re:Intel SA-00145: Lazy FP State Restore by techntoke on Wednesday June 13, 2018 @06:58PM (#17307738)
Does that include Red Hat Enterprise Linux? I can't believe how old of a kernel they use still. I have to use it for work, but I run Arch at home which is running 4.16.13. I'm honestly surprised that Red Hat can't manage to keep their distros current with stable package builds.
Re:Intel SA-00145: Lazy FP State Restore by snuxoll on Wednesday June 13, 2018 @07:09PM (#17307813)
Red Hat backports security fixes, hardware support and certain new features to their kernel packages, staying on a stable kernel version throughout the product lifecycle is what allows them to guarantee a consistent kernel ABI that virtually every other distribution throws to the wayside. Personally I prefer not having third-party kernel modules on my servers break every time I run `yum upgrade` like I do with Fedora (oh VMWare, how I hate you so).
Re:Intel SA-00145: Lazy FP State Restore by readams on Wednesday June 13, 2018 @07:14PM (#17307837)
except they still break because Red Hat backports bleeding-edge features but doesn't change the version number.
Re:Intel SA-00145: Lazy FP State Restore by vbezhenar on Wednesday June 13, 2018 @07:29PM (#17307932)
I'm using HP B120i fakeraid controller with proprietary driver and it broke after 7.4 upgrade, so while they probably doing a good job about binary compatibility (7.5 didn't broke it), it's not ideal.
Re:Intel SA-00145: Lazy FP State Restore by snuxoll on Wednesday June 13, 2018 @11:16PM (#17308988)
Unfortunately the kABI does not encompass every symbol exported by the kernel, there is a whitelist maintained in the kernel-abi-whitelists package and scripts to check conformance of a module to the whitelist. Symbols are only ever added throughout the lifecycle of a RHEL release, so anything that conforms to the 7.3 kABI will also work on 7.4 - but if the module uses a symbol NOT whitelisted in 7.3 there's no guarantee the 7.4 update won't break it.

Not that it's much solace when your proprietary blob breaks, but Red Hat does make considerable effort to give vendors a stable target to build against - unfortunately not everyone fully validates conformance which results in the same old problems cropping up from time to time.

Re:Intel SA-00145: Lazy FP State Restore by silly-silly on Thursday June 14, 2018 @05:48AM (#17310510)
That and Spectre fixes forced a BUNCH of third party code that relied on non exported code to break.
Re:Intel SA-00145: Lazy FP State Restore by xnyanta on Thursday June 14, 2018 @09:12AM (#17311383)
I really hate HP for this stupid fakeraid shit. The regular SATA AHCI controller is unusable on the HP servers I have because it does not pass the temperature info to the iLO and causes the fans to skyrocket to 70% permanently.

Obviously, using the B120i was out of question due to their crappy driver only targeting RedHat's ancient kernel so my only viable solution was to have to go out and purchase a 200$ real HP P420 smart array RAID card to connect my drives to.

Re:Intel SA-00145: Lazy FP State Restore by 4ad on Thursday June 14, 2018 @05:43AM (#17310490)
> I'm honestly surprised that Red Hat can't manage to keep their distros current with stable package builds.

This is very wrong.

It would be trivial for Red Hat to update the kernel shipped into RHEL. It's not a question that they "can't keep up".

What they are doing is much more difficult and takes much more effort. They are backporting all security fixes, and sometimes other features from newer kernels to older kernels. Why? Because many RHEL customers want older kernels. Unlike Windows and Solaris, Linux doesn't have a stable kernel ABI (by design). Some RHEL customers want to use binary drivers, and they want their drivers to work when they update their system. Some other RHEL customers do not use or need binary drivers, but they do want to avoid possible incompatibility problems introduced by newer kernels. And no, it doesn't matter that the kernel API is very stable. Sometimes even fixing bugs can introduce incompatibility problems in badly written software. Other times something like changing some trivial subtle thing about the scheduler can make a badly written program perform much differently then it used to be. The world is full of such programs, and people who administer such systems are very grateful the behavior of their systems doesn't randomly change across the lifetime of a release.

The fact that RHEL freezes the versions of shipped software is its greatest feature. It's not for everyone, but it's for some people, and for those people it is very important.

Re:Intel SA-00145: Lazy FP State Restore by burb_xry on Thursday June 14, 2018 @07:17AM (#17310788)
The problem with that is that I'd argue it is "safer" to run a kernel.org stable kernel rather than RHEL, since it's sheer impossible to keep up with the amount of fixes that eventually land in stable kernels every few weeks. And it is unclear which of the stable fixes are security relevant ones even if they did not get a CVE number and the like. QAing all these is another mammoth task that often requires expertise in several areas. I'm actually glad that (upstream) Linux doesn't have a stable kernel ABI, if you've ever looked into what gross hacks are required in the RHEL kernel only to keep that intact, well, good luck.
Re:Intel SA-00145: Lazy FP State Restore by kjeetgill on Thursday June 14, 2018 @12:55PM (#17313105)
That's not the choice customers have. The parent gave examples for why companies often need predictability and stable ABIs. They're choosing between no kernel upgrades and RHEL's.

What good is a fresh kernel if a binary driver you need but can't change breaks?

Re:Intel SA-00145: Lazy FP State Restore by techntoke on Thursday June 14, 2018 @03:24PM (#17314586)
Right.. it's like they haven't managed to automate a CI pipeline to handle these changes for them. Other distros do it just fine, and I've had far more things break in CentOS or Red Hat than Arch.
Re:Intel SA-00145: Lazy FP State Restore by slrz on Thursday June 14, 2018 @10:04PM (#17317000)
That CI pipeline would need to be placed not at Red Hat but at their big customers and their hardware vendors. To RH's benefit, those tend to suck at that and prefer to pay big dollars over having to keep up with kernel development or getting rid of their crappy out-of-tree drivers.

If you have the appropriate CI pipelines as well as the kind of developers capable of directly consuming the upstream community outputs, by all means, do that. Many companies aren't like that though and prefer talking to tech support people over diving into the source code themselves. I guess we should just be happy about the fact that they fund Linux development at Red Hat.

Re:Intel SA-00145: Lazy FP State Restore by cesarb on Thursday June 14, 2018 @07:05AM (#17310745)
> Other times something like changing some trivial subtle thing about the scheduler can make a badly written program perform much differently then it used to be.

Like changing the scheduler preferences of which process runs first on fork(), the parent or the child, exposed a bug in bash: https://yarchive.net/comp/linux/child-runs-first.html

Re:Intel SA-00145: Lazy FP State Restore by gicmo on Wednesday June 13, 2018 @07:09PM (#17307804)
RHEL kernels are heavily patched, i.e. stuff gets backported from new kernels, sometimes even from -rc if needed. The version number is therefore not that useful. Some call it a Frankenkernel. :)
Re:Intel SA-00145: Lazy FP State Restore by kbenson on Wednesday June 13, 2018 @10:24PM (#17308759)
The nice thing is that it's all fully documented. If you download the kernel src.rpm[1] and install it, you'll get the kernel RPM spec file in your default RPM build environment paths (or use rpm2cpio $KERNEL_SRPM | cpio -idmv). The way the RPM building works is to take a vanilla kernel and then apply a LOT of individual patch files to it, each with their own entry.

Well, at least that's how it was a long time ago when I messed with this stuff more, but it looks like it may be a bit different now (I just did what I proposed), at least with the downstream CentOS kernel. I stopped having to care about this mostly back around the time when Oracle forked RHEL/CentOS into Oracle Linux and RedHat wasn't happy about Oracle piggybacking on their kernel testing for a paid product (as opposed to CentOS). I think maybe RedHat ships the kernel tar mostly pre-patched in-house now. I may have some of the details wrong in that, but it sounds right to me. It's easy to blame Oracle for why we can't have nice things anymore, because usually it's true. :/

Edit: Ah, I found one like what I was talking about[2], it has 5697 patches included! The most recent 5 series kernel (5.11) didn't have a lot of patches, but 5.6 did. There might be some 6.x series ones like that as well, I don't recall.

1: http://vault.centos.org/centos/7/os/Source/SPackages/kernel-...

2: http://vault.centos.org/5.6/os/SRPMS/kernel-2.6.18-238.el5.s...

Re:Intel SA-00145: Lazy FP State Restore by makomk on Thursday June 14, 2018 @04:27AM (#17310244)
Yeah, as of a few years ago Red Hat only give access to the proper broken-out set of patches they include in their kernel to paying customers, and they make those customers sign a contract agreeing not to distribute those patches. (They're also the only major distro who doesn't cooperate with the upstream stable releases.)
Re:Intel SA-00145: Lazy FP State Restore by kevin_thibedeau on Thursday June 14, 2018 @10:09AM (#17311756)
How is that not a GPL violation?
Re:Intel SA-00145: Lazy FP State Restore by kbenson on Thursday June 14, 2018 @02:53PM (#17314329)
The source code is released, it's just the formatting that has changed. Presumably the contract is around the extra effort RedHat puts into making the source nice and easily manageable by providing what they used to by default, a pristine kernal and a bunch of patches to apply.

The normal method now is that they provide a large, pre-patches kernel source, which is then built. In both cases, the same code source is used for building,it's just the transformations before the building which are different, so that's likely how it doesn't violate the GPL.

The GPL says the source must be shipped, but it makes no provisions for how easy it is to read. It's sort of like if you shipped a JS lib as minified to comply with the GPL, but have a separate non-minified version to develop on in-house.

In the end, it doesn't stop anyone from downloading and compiling their own RedHat kernel as CentOS does, and Oracle Linux did(does?), but it does make it harder (not impossible) for Oracle to come in to RHEL customers and say "hey, stop your RHEL contracts, and pay us instead, and we'll support your systems without you having to reinstall." which is capitalizing on RedHat's packaging and testing work.

Re:Intel SA-00145: Lazy FP State Restore by lathiat on Friday June 15, 2018 @07:27AM (#17318961)
Oracle then started a project to break them back out again. Last update seems to be November 2017 though.

https://oss.oracle.com/projects/RedPatch/

Re:Intel SA-00145: Lazy FP State Restore by danieldk on Thursday June 14, 2018 @02:22AM (#17309714)
I think maybe RedHat ships the kernel tar mostly pre-patched in-house now.

Yes, this was in the news a couple of years ago. They do not include separate patches anymore:

https://lwn.net/Articles/430098/

https://www.theregister.co.uk/2011/03/04/red_hat_twarts_orac...

Re:Intel SA-00145: Lazy FP State Restore by pera on Wednesday June 13, 2018 @08:43PM (#17308329)
> RHEL-7 will automatically default to (safe) “eager” floating point register restore on Sandy Bridge and newer Intel processors. AMD processors are not affected.

https://access.redhat.com/solutions/3485131

Re:Intel SA-00145: Lazy FP State Restore by Twirrim on Wednesday June 13, 2018 @09:26PM (#17308532)
It looks like for older processors they may be vulnerable?

https://access.redhat.com/security/cve/cve-2018-3665 shows there will be updated kernels coming.

Re:Intel SA-00145: Lazy FP State Restore by hinkley on Wednesday June 13, 2018 @08:25PM (#17308230)
Contrast this with the Postgres addon community, which has a culture of pushing changes upstream until they no longer have to maintain a fork of the Postgres code.

Build a business on a fork, then compartmentalize it until it can run without modifying the core so you can get off the bugfix treadmill.

Re:Intel SA-00145: Lazy FP State Restore by dralley on Wednesday June 13, 2018 @10:05PM (#17308683)
The changes are all upstream to begin with.

It's a fork in the sense that it's heavily diverged from the source of the old kernel, but it's all still backports and not a ton of new development.

Re:Intel SA-00145: Lazy FP State Restore by hinkley on Thursday June 14, 2018 @02:07AM (#17309667)
Why does it need to be diverged if there is no new functionality in their version? Because they can?
Re:Intel SA-00145: Lazy FP State Restore by kikoreis on Thursday June 14, 2018 @04:56AM (#17310349)
ABI stability to support binary-only loadable modules is probably the most important reason.
Re:Intel SA-00145: Lazy FP State Restore by vertex-four on Thursday June 14, 2018 @03:39AM (#17310063)
Why do many distros have releases to which only bug and security fixes are applied? Same reason. The divergence is just those fixes.