News for hackers. Stuff that matters.
Re:Finding a kernel regression in half an hour with git bise... by dekhn on Wednesday July 28, 2021 @05:58PM (#27990276)
We used to have to do this a lot on my team at Google, except with Google tools (piper and blaze). The difference is that because Google is a monorepo, you often end up with 50K+changes to sort through, at which the log2 propertyy of bisection becomes necessary, not just convenient.
Re:Finding a kernel regression in half an hour with git bise... by ComputerGuru on Wednesday July 28, 2021 @07:01PM (#27990857)
I’m sure you already know, but can you restrict it to changes from one or more directories to avoid that, worst case scenario it’ll point you to the general change even if it was due to another repo necessitating the migration.
Re:Finding a kernel regression in half an hour with git bise... by dekhn on Wednesday July 28, 2021 @07:15PM (#27990966)
i worked with code that could be broken by a wide range of things, anything from a change in tensorflow, XLA, device drivers, firmware, a support library, to cluster software upgrades that happens async (sometimes months after checkin) from changes in the repo.

I used to update a major numerical python library periodically, and sometimes the global test would show that some Java code broke. at first, I assumed it was just a false positive, but some sleuthing showed the Java code had embedded a python script that ran numpy to compute something over an array (whyyyyyyyyy) and numpy just happened to start outputting some slightly different values for that computation.

These are the perils and promises of the monorepo.

Re:Finding a kernel regression in half an hour with git bise... by mey on Wednesday July 28, 2021 @08:52PM (#27991549)
Out of curiosity, was there a process to raise that design "oddity" to someone? Is there a way in the Google mono-repo to even know who is responsible for which section?
Re:Finding a kernel regression in half an hour with git bise... by iamstupidsimple on Wednesday July 28, 2021 @08:54PM (#27991560)
There are 'OWNERS' files which list email addresses for directories, but anyone can contribute any piece of code. Generally, it is like a giant open source project to the eyes of a developer.
Re:Finding a kernel regression in half an hour with git bise... by dekhn on Thursday July 29, 2021 @09:00AM (#27995969)
yes you can use the 'blame' feature to identify specific lines, or look in piper to see who edited a directory recently. In addition to OWNERS files.

By the time I finished maintaining numpy, I could more or less find the right person with a couple minutes poking. Again, it's one of the things that google managed to do well- a large scale code base with many developers working more or less in concert, with efficiency at scale.

Re:Finding a kernel regression in half an hour with git bise... by joshuamorton on Thursday July 29, 2021 @04:28AM (#27994171)
You can still do the sneaky thing to query the build graph and only get changes in the transitive dependencies. This is almost always far smaller than the entire range between good and bad. For fast tests, the cost of doing the dependency lookup can be not worth it, but if your bisection will take more than 10 minutes, it's probably worth the effort.

It's still a little more complicated because then you have to bisect over a list of changes and not a raw range, but still.

Re:Finding a kernel regression in half an hour with git bise... by kazinator on Wednesday July 28, 2021 @07:26PM (#27991035)
log2 bisection is pretty necessary for just hundreds or even dozens of commits. I'm not linearly searching through the history of something that takes minutes to build and requires manual installation and test steps.
Re:Finding a kernel regression in half an hour with git bise... by fragmede on Thursday July 29, 2021 @12:53AM (#27992887)
Automating the installation and test, and then pointing `git bisect run script` at it is pretty damn awesome if you can automate it.
Re:Finding a kernel regression in half an hour with git bise... by bspammer on Thursday July 29, 2021 @04:42AM (#27994258)
This is one of my absolute favourite things to do. There’s something so beautiful about having software automatically finding the bug for you.
Re:Finding a kernel regression in half an hour with git bise... by globular-toast on Thursday July 29, 2021 @11:07AM (#27997417)
This is why keeping good, linear history is so important. The software isn't magic. It won't be able to tell you where the bug is in that giant commit that touched every module in the codebase. Good history is your input to git bisect. It's the only reason (but an extremely compelling reason) to care about your git history.
Re:Finding a kernel regression in half an hour with git bise... by kazinator on Thursday July 29, 2021 @06:36PM (#28002316)
Here is a bisect scenario that seems to be happening before my eyes.

I was using git bisect to find a problem. Let's call it issue A. During the bisect, I discovered that some builds are not testable at all. Let's assume that instance of that problem are all issue B. So the verdict from the git bisect is that there are four possible consecutive commits that could be the start of A.

But those four include commits afflicted with B, which seems related. Why it seems related is that the last of those four bad commits suddenly fixed B, so builds became testable, but introduced A.

Since I was bisecting for A, I don't have a complete list of commits afflicted with B; git bisect just happened to find three commits afflicted with B.

Thus I have to do a second git bisect, this time validating for issue B, to see where exactly the builds became untestable.

That will hopefully put together the story: something like: at commit X, a serious problem showed up, rendering builds untestable (issue B). Then at later commit Y, B was fixed but A started happening, which wasn't a problem before X.

Re:Finding a kernel regression in half an hour with git bise... by kazinator on Thursday July 29, 2021 @02:17PM (#27999655)
Well, find the commit where the behavior is introduced, in any case.

Or, in some cases, multiple possible commits. If the bad commit is preceded or followed by untestable commits (declared that way via "git bisect skip", they are implicated also).