News for hackers. Stuff that matters.
Re:Teardown of a Failed Linux LTS Spectre Fix by sdrothrock on Wednesday September 04, 2019 @05:19AM (#20875393)
> The takeaways here are that there is a real benefit to having an external/independent review and backporting process like the one we perform for our customers

That's great for sales for these people, but I think the real takeaway here is:

> when the actual merge of the tree was performed, no mention was made of [Linus's] correction to the [original] fix, and with no specific commit mentioning the correction and fixing it alone, everyone else's processes that depended on cherry-picking specific commits ended up grabbing the bad warning-inducing change. As a further failure, instead of looking at Linus' correct fix (observable by checking out the master tree at the time), the approach employed in the LTS kernels seems to have been to naively silence the warning

There are a LOT of process failures in this description of what happened. At the kernel development level, that should be very concerning.

It's good that an external auditor found it (hooray open source), but this was almost definitely an easily preventable mistake with a better process.

> everyone else's processes that depended on cherry-picking specific commits

This alone sounds horrifying.

I don't know anything about kernel development, so I don't want to sit here and judge and offer advice out of ignorance, but I really do feel like there's a better process that would work for them that doesn't involve people "cherry-picking specific commits."

Re:Teardown of a Failed Linux LTS Spectre Fix by richardwhiuk on Wednesday September 04, 2019 @05:35AM (#20875463)
Can you make a recommendation of a process which doesn't involve cherry-pick commits (or more often patch-sets)?

Note, your process needs to allow a fix to be developed against V5.1, and back-ported to V4.1.

I suspect one of the problems here is a lack of bug tracker, which would have associated the fix to the merge and the fix itself together.

Re:Teardown of a Failed Linux LTS Spectre Fix by sdrothrock on Wednesday September 04, 2019 @05:40AM (#20875487)
> Can you make a recommendation of a process which doesn't involve cherry-pick commits (or more often patch-sets)?

Thank you! You just made me realize that there some ambiguity in the original article and thus in my reply; I have nothing against cherry-pick in git and first thought the original article was using the phrase idiomatically, as in "developers picked specific commits with their personal judgment."

The rest of this response will be written with that idiomatic reading in mind since that's what I originally intended with my criticism.

> I suspect one of the problems here is a lack of bug tracker, which would have associated the fix to the merge and the fix itself together.

I agree with this. This would make it immediately obvious that there were multiple fixes for one issue, which should be strange.

I also think there should be an overall review process where both:

1. Commits that have been chosen are reviewed by the group

2. Commits that have NOT been chosen should be reviewed by the submitters -- if there's something critical being left out, this is the time to speak up

They may already be doing something like this, for all I know, and there may be other reasons that this happened.

Additionally, I'm only coming to this conclusion because of the way it was presented in the article; the people involved may have a different view of why this happened entirely.

But for me, the way it was presented in the article, it definitely looks like a process failure.

Re:Teardown of a Failed Linux LTS Spectre Fix by Jach on Wednesday September 04, 2019 @10:53AM (#20877716)
You can only have so much process before you're too far out of the bazaar. This isn't to bring up the relative value tradeoffs of that old split, just to highlight that one shouldn't expect a 'process adjustment' reaction from a fundamentally anti-process collective or for members to even view it primarily through the lens of a process failure. (I'd sooner expect a rant on basic C coding competence even though those days are gone.) That's even if the LKML reflects on this particular timeline of patches much at all; there's no good reason to, really, it's a curiosity. And unlike those of us in a proprietary enterprise no one expects or can insist on an RCA doc with promises to do better backed by planned process tweaks (what issues can't ultimately be recast as process failures?) or new processes.
Re:Teardown of a Failed Linux LTS Spectre Fix by Nursie on Wednesday September 04, 2019 @01:55PM (#20879711)
When I was a developer on some enterprise software, our pattern was that while features were developed against the newest codebase, bug-fixing was done against the oldest supported version that exhibited the bug, and then rolled forward from there to all other releases.

This avoided the cherry picking entirely.

(I understand this wouldn't be a good fit for linux kernel development, but it is an approach I've seen work well)

Re:Teardown of a Failed Linux LTS Spectre Fix by the8472 on Wednesday September 04, 2019 @10:59AM (#20877770)
Maybe linus' fix should have been added as separate commit to a branch instead of putting the fix in the merge commit itself? Then the entire branch could be taken and rebased, or at least the range cherry-picked.
Re:Teardown of a Failed Linux LTS Spectre Fix by stonesixone on Wednesday September 04, 2019 @01:39PM (#20879546)
Indeed, this is called an "evil merge" [1], and here's [2] even an email with Linus mentioning it. I'm surprised the blog post doesn't mention evil merges by name, as this seems to be a prime example of why they're called evil.

[1]: https://git-scm.com/docs/gitglossary#Documentation/gitglossa...

[2]: https://www.mail-archive.com/[email protected]/msg73938.ht...

Re:Teardown of a Failed Linux LTS Spectre Fix by 0x0 on Wednesday September 04, 2019 @12:34PM (#20878835)
That's what I was thinking too. Extra diff changes in a merge commit is often hard to notice
Re:Teardown of a Failed Linux LTS Spectre Fix by CalChris on Wednesday September 04, 2019 @01:35PM (#20879486)
> Can you make a recommendation of a process which doesn't involve cherry-pick commits (or more often patch-sets)?

This is for LTS, Long Term Support. Why would the maintainers be cherry picking candidate patch sets at all? Shouldn't they wait for a release and back port from that? Regular release intervals are 8-10 weeks.

That would be my recommendation, backport only from releases. This is especially the case here because it was a security fix.

Re:Teardown of a Failed Linux LTS Spectre Fix by cyphar on Wednesday September 04, 2019 @02:22PM (#20879976)
Patches have to be in Linus's tree before they land in -stable. That's been the policy for at least as long as I remember. Also, Greg does releases of -stable every week or two -- waiting 8 weeks for patches (that Linus has already merged) makes very little sense.

Yes, fixup patches are something that need to be handled separately but Greg has always handled those properly (and good kernel devs use the Fixes: tags and always Cc: stable on those fixup patches). Not to mention you'd need to handle fixup patches even if you did depend on releases (if a bug was introduced in a released version it's usually fixed by the next -rc1 -- so waiting means the newest kernel is broken needlessly for 8 weeks).

Re:Teardown of a Failed Linux LTS Spectre Fix by SolarNet on Wednesday September 04, 2019 @02:04PM (#20879808)
Because a significant portion of those changes will conflict, and one will have to go back to each commit that made the changes to figure out the intention of the change.

Easier to back port each set of changes while knowing what the point of the changes are.

Re:Teardown of a Failed Linux LTS Spectre Fix by richardwhiuk on Friday September 06, 2019 @04:26AM (#20893983)
How would you back-port from a release, apart from cherry-picking commits or patch-sets?
Re:Teardown of a Failed Linux LTS Spectre Fix by Dylan16807 on Wednesday September 04, 2019 @06:35AM (#20875765)
> the approach employed in the LTS kernels seems to have been to naively silence the warning

I wouldn't even give it that much credit. Putting "line that assigns to var" after "line that uses var" is worse than 'naive'.