News for hackers. Stuff that matters.
Re:Microsoft hits out at Google team over bug report by Beltiras on Monday January 12, 2015 @09:00AM (#8873843)
2 days leeway is not uncalled for. Bad form Google.
Re:Microsoft hits out at Google team over bug report by spydum on Monday January 12, 2015 @09:11AM (#8873899)
unlikely we will know the true internal discussion, but it seems unlikely it realistically took MSFT 92 days to put together a patch.

It seems far more likely they sat on the disclosure and did not prioritize it appropriately, after google continued to push for an estimated release date (which they never got). Then when the clock was running out, MSFT asked for extension, and Google being tired of getting put off, said no and stuck to their 90-day policy.

all of the above is speculation, and I don't necessarily agree it was the right thing to do. However, I could see folks @ googles sec group being a bit pissed MSFT wasnt taking it seriously, and then decided to stand on their policy just to make an example (surely, MSFT wont drag feet on future disclosures after this, right?).

Re:Microsoft hits out at Google team over bug report by cmdkeen on Monday January 12, 2015 @09:33AM (#8874024)
Alternatively, and without your unsubstantiated knocking of Microsoft, perhaps they were needing to conduct serious regression testing. This isn't the "unit tests pass, whack it into production" leagues after all.

Plus there's also the small matter of Christmas, and for many countries, significant developer unavailability falling into that period.

What's more likely, Microsoft can produce, test and deploy a patch in a few days, but chose not to until they had to. Or that they did get a patch ready and tested, but not in time for the patch Tuesday the previous month (so actually got it done in ~62 days) but it didn't warrant an out of band release?

Re:Microsoft hits out at Google team over bug report by orf on Monday January 12, 2015 @09:54AM (#8874130)
The run tests over every single supported version of Windows, on every platform and with every combination of patches installed. Or so I read in an article by a MS employee who deals with security reports, someone sent in a nonsense 'vulnerability' with some C# POC involving MessageBox.Show and returning from main. It was obviously not in any way a issue, but he still ran it across all versions and patch-levels of windows to be sure. It doesn't say how long this took but I expect it wasn't fast.
Re:Microsoft hits out at Google team over bug report by doodpants on Monday January 12, 2015 @10:00AM (#8874164)
Arguably, the fact that Google was going to reveal the vulnerability two days before the next patch Tuesday could have been considered sufficient reason to warrant an out of band realease.
Re:Microsoft hits out at Google team over bug report by ethomson on Monday January 12, 2015 @11:02AM (#8874587)
I very much doubt that this was simply "sat on". Not knowing the full breadth of this vulnerability, I can only speculate, but I'm making an educated guess based on several years of working at Microsoft and recently managing a fix for a security vulnerability in several versions of two products.

It can sound crazy to you (certainly, it sounded crazy to me at first), but taking 90 days to turn around a fix to a piece of software like Windows is not implausible.

First you have to understand the vulnerability: under what circumstances does it occur? Can we build some reliable tests so that we are convinced that we will know when we have fixed it?

Looking at the report of the vulnerability, are there other instances of problems that will lead to similar bugs? Because if you release a patch and then a week later somebody realizes a similar vulnerability, you're going to have to do all of this over again, and not with the luxury of 90 days before the announcement.

Then you have to identify what versions of the software are affected. This is where you start sweating, because were you working on the product seven years ago? If you were, do you remember anything about it? Well, you're going to have to learn fast and hope that the architecture hasn't changed too much.

Then you fix the bug, probably only in one of the affected versions at first, which is probably the version of the software that you have on your dev box. As you're fixing it, poke around in the code to think about similar vulnerabilities that the report didn't find. Recall that Windows is not a small piece of software and building it on your dev box and being able to test your fix may take several hours.

Now you port that bug fix over to all the other versions of this software that are affected and that you still support. For a piece of software like Windows, this is a long list. Hope that the architecture and the code you're fixing hasn't changed much or else you're digging in to remember how Windows 2008 worked and how to fix this problem there. Recall again that building each of these versions takes time.

Now you hand off your fixes to some other people who will independently verify your fixes on some of the supported versions. I say some, not all, because there's going to be another round of testing on the actual deliverables, which is the patch to the operating system.

Which another group is going to make. Most products at Microsoft have a build lab that will take a branch and create either the actual installer disk image or a patch to a previous version. In this case, they're going to create a patch to the latest supported version, for each supported version.

Fortunately, this can probably be done in parallel with the first round of testing if you're pretty sure that you nailed it. If you think that the testing (above) is likely to reveal some problem, then you should hold off handing it over to the build lab because these folks are some of the least appreciated parts of the development team. They're the ones who integrate all the various development teams feature branches into master, resolve the easy conflicts and find the people who need to resolve the hard ones. They have a full time job (and not a trivial one) before you're bringing your high priority build to them, and when you ask them to dust off the build machines for a seven year old version of the product, they're going to graciously accept. But to tell them you didn't get it right and request they start over on a new version is when you start bringing six packs with your request.

Once the patch is created, it goes through the real testing. Because somebody's going to install this patch on all the supported versions, and the SKUs within those versions, to make sure that it works. When I say "it works", I don't just mean that the patch fixes the bug in question (though of course it has to do that), I mean that it also has to not regress any functionality. And that the patch is able to be installed and uninstalled cleanly. This is some annoying work and the longer this bug has been around, the more annoying it is. Remember "Windows Essentials Business Server 2008"? Me neither, but somebody's going to be installing it on a VM and ensuring that your patch works there.

Let's assume that everything has gone well up to this point and you're ready to release it. Most product updates want to get included in Windows Update, of course, because you want your security fixes to just show up to the customer without them having to learn about them, download it and install it because so few people do.

If you're going to miss a patch tuesday, then you need to start asking yourself whether you want to a) try to buy yourself some more time, b) put up several KBs with the patch and ask people to install them manually or c) wait it out until the next patch tuesday. What you decide will probably be some combination of those depending on the severity of the bug. There's a possible fourth option, which is to convince somebody that your patch needs to go out before patch tuesday and while I'm sure that happens, I'm not sure how it happens. I suspect when you're dealing with a bug so critical as to warrant that level of pain for the organization, it will happen.

All of these steps take time. And a big organization like Microsoft moves slowly sometimes - it takes time to find the right people for all of these steps. This is especially difficult over the holidays where many of the "right people" here are out of the office.

Now you could argue that it's ridiculous that it took Microsoft 92 days to get a fix out the door, and maybe you'd be right. But that's another thing entirely. What's clear to me that Microsoft did not simply wait until day 89 to start working on this.

(In relating this story, I would be remiss if I didn't thank Junio Hamano of Google, the maintainer of git, who was kind enough to provide Microsoft with additional time to research and prepare patches for the range of our products that were affected by CVE 2014-9390.)

Re:Microsoft hits out at Google team over bug report by ethomson on Monday January 12, 2015 @11:12AM (#8874657)
Despite the long wall of text I just posted, I forgot to mention: some clever person is inevitably going to come at the end of this whole process and say "oh hey guys? I figured out another way to trigger this bug" and you start over from the beginning.
Re:Microsoft hits out at Google team over bug report by spydum on Monday January 12, 2015 @12:23PM (#8875060)
start over? the 90 days is not spent attacking the code.. it's split across testing/packaging/etc. The developers can continue to pour over the code in PARALLEL to the testing to check for similar avenues/triggers, and at any point, push an updated patch, which could move the release date back. Still, how long would it truly take to finish the full cycle? It seems hard to believe it would be months. 90 days is roughly 12+ WEEKS.

Let's take a stab in the dark and say it takes 4 weeks to fully test. That means from day one, MSFT would have 8+ weeks of time for developers to attack the problem (yes blah blah patch tuesday, call it 7 weeks if you wish). I don't know their code, and I don't know how pervasive the problem was, but 7 weeks seems pretty generous when you are handed the problem (and not required to fish it out of "user" complaints).

Re:Microsoft hits out at Google team over bug report by ethomson on Monday January 12, 2015 @12:52PM (#8875267)
Right. Like I said, it sounds crazy that anything would ever take this long. And I'm not saying that it should take 90 days to get a patch out the door. I'm saying that it plausibly take 90 days to get a patch out the door.

For CVE 2014-9390, we had a fix roughly 30 days after we were notified of the vulnerability. In that time, we released patches for the third-party libraries that we use for Git repository management (libgit2 and LibGit2Sharp), we released patches to three versions of Visual Studio and Team Foundation Server, and we worked with the Git for Windows team to ensure that our fixes were nice and compatible. In order to get the VS and TFS patches out the door, we needed all of this time.

(In fact, we discovered additional vulnerabilities on Mac OS very late in the process. Had these problems affected Windows, too, we would have needed to throw our patches away and start over with the build / test validation process.)

And compared to Windows, Visual Studio is not nearly as big or complex, and can turn these changes around faster.

Re:Microsoft hits out at Google team over bug report by nodata on Monday January 12, 2015 @09:04AM (#8873856)
Didn't they give MS 90 days to patch? (That's two or three patch Tuesdays).
Re:Microsoft hits out at Google team over bug report by _wmd on Monday January 12, 2015 @09:38AM (#8874045)
As another comment points out, unlike Google updates, Microsoft's occur on a particular day of every month. This allows end user IT departments to coordinate the updates as part of a predictable change management policy.

In this case, the bug was reported on 13 Oct, 15 working days before the first Tuesday of November. Assuming Microsoft couldn't fix and test a change to a core service running on millions of desktops in that time, their only remaining opportunity within the disclosure window was the patch day in December - allowing them only 50 calendar days to deal with the bug.

What's worse is that the bug isn't even all that severe. Effectively Google were trying to strong-arm Microsoft into releasing an out of band patch for a minor issue, which would have knock-on effects for thousands of IT departments. A completely dick move, and thankfully one that's being given recognition here.

Shit like this is why I have a hard time dealing with the infosec community in general – a strange mix of conformance to pointless minutia, a deeply ingrained sense of self-importance (only worsening over time with PR crap like APT), and a fatal attraction with some of the most puerile elements of society. The result is a unique melting pot of genius and dumb that I regularly can't stomach.

Re:Microsoft hits out at Google team over bug report by majc2 on Monday January 12, 2015 @09:10AM (#8873890)
And they asked for 92 days and Google decided to publish anyway? While I'm not surprised, it doesn't strike me as the most responsible action.
Re:Microsoft hits out at Google team over bug report by tomiko_nakamura on Monday January 12, 2015 @09:31AM (#8874011)
If you define a limit for disclosure, and then not stick to it, why to define a limit in the first place? 90 days is more than enough - if MS has a lot of internal overhead, you should probably complain to them, not to google.
Re:Microsoft hits out at Google team over bug report by Beltiras on Monday January 12, 2015 @10:20AM (#8874259)
If contacted by the other party and they give a good reason (in this case: "We have a fix, it's slated for release in line with other things on tuesday"), I think a responsible security researcher should give that time. If patch day rolls around and no production, go ahead and shame. This is not a case of overhead, MS world functions a bit differently from package management in Linux.
Re:Microsoft hits out at Google team over bug report by majc2 on Monday January 12, 2015 @10:15AM (#8874232)
"Our objective is to significantly reduce the number of people harmed by targeted attacks." (http://googleprojectzero.blogspot.co.uk/2014/07/announcing-p...)

They've lost sight of this noble objective with an inflexible policy; who anointed Project Zero guardians of the internet? Why not wait the two days? cui bono?

Re:Microsoft hits out at Google team over bug report by shoggs on Monday January 12, 2015 @09:09AM (#8873886)
Apparently Microsoft needed some time for testing.
Re:Microsoft hits out at Google team over bug report by pilif on Monday January 12, 2015 @09:10AM (#8873889)
If Google insists on exactly 90 days without any consideration for patch Tuesdays, this means that, if unlucky, MS will only get 60 days (if they got the announcement right after a patch Tuesday)
Re:Microsoft hits out at Google team over bug report by tomiko_nakamura on Monday January 12, 2015 @09:28AM (#8873986)
So, who forces Microsoft to stick to (so called) patch Tuesdays? No one, actually - it's Microsoft internal schedule, and clearly there are cases when it's absolutely unreasonable - e.g. when there's a 0day in the wild. So there has to be a way to fast track a fix - if there's not, there's something seriously wrong IMNSHO. Apparently, they thought Google won't stick to the 90-day limit.
Re:Microsoft hits out at Google team over bug report by Someone1234 on Monday January 12, 2015 @10:00AM (#8874162)
> So, who forces Microsoft to stick to (so called) patch Tuesdays?

Their tens of millions of customers who plan internal deployment, overtime, and other things around those specific dates?

> it's Microsoft internal schedule

It's their external schedule actually.

Re:Microsoft hits out at Google team over bug report by jdjb on Monday January 12, 2015 @11:46AM (#8874838)
Why do all these customers need to be on Microsoft's schedule though? Microsoft should release the patches and give each IT team their own ability to assess patches at their own speed.

If some companies want to wait 2 days then let them make their own choice. It seems like a pretty stupid policy that nothing (except extreme cases) should get patched except when it's convenient.

Re:Microsoft hits out at Google team over bug report by Someone1234 on Monday January 12, 2015 @11:54AM (#8874887)
> Microsoft should release the patches and give each IT team their own ability to assess patches at their own speed.

They already do via WSUS. Companies get to plan this work to start on the second Tuesday of every month because that's the day Microsoft publishes them. Nobody puts a gun to company's heads and forces them to deploy internally.

> If some companies want to wait 2 days then let them make their own choice. It seems like a pretty stupid policy that nothing (except extreme cases) should get patched except when it's convenient.

This seems to be a complaint about a "policy" which doesn't exist and has no relationship to the topic at hand. I don't even really entirely understand the above.

Re:Microsoft hits out at Google team over bug report by tomiko_nakamura on Monday January 12, 2015 @03:08PM (#8876114)
I don't really see how that matters. This only shows how utterly broken the concept of "patch Tuesdays" is ...

If you plan your internal deployment updates based on the belief that the schedule will never change, then I'm really sorry for you and your users. In real world, not all issues are reported in advance - some are observed in the wild, and in that case you have to fast-track the fix. If you have no way to do that (e.g. because the vendor only releases fixes on Tuesdays once per month, or because you decided to choose such schedule on your own), then good luck. That might have been appropriate in 1995, not in 2015.

There are many projects and/or companies publishing fixes continuously, and leaving it up to the users when/how to apply them in production. That's essentially what all the linux distributions (RH, Suse, ...) and smaller projects do.

Re:Microsoft hits out at Google team over bug report by matwood on Monday January 12, 2015 @10:12AM (#8874214)
it's Microsoft internal schedule

Not entirely true now. Users of MS software have built up their own testing processes around patch Tuesday.

Patch Tuesday was one of the best things MS did when they decided to take security seriously. They realized that testing patches downstream takes time and giving their customers a consistent patch day let them also plan ahead.

Re:Microsoft hits out at Google team over bug report by gizmo686 on Monday January 12, 2015 @09:24AM (#8873966)
The reason for giving 90 days is not that some issues are so complicated that they require 90 days to fix, but that bureaucracies can be slow and need a lot of extra time. In your case, that is 30 days of bureaucratic overhead, leaving them 60 days of actual work. This seems reasonable, as bussiness deadlines are 30 days.