News for hackers. Stuff that matters.
Re:We found and fixed a rare race condition in our session h... by mcherm on Thursday March 18, 2021 @04:57PM (#26507482)
I just want to make clear that for me, write ups containing this level of detail analyzing the error that occurred give me greater confidence in the company rather than less confidence. Greater even than if I had never heard of the original bug. There are many bugs I have never heard about, but any company that uses this level of care and openness is likely to be very reliable.
Re:We found and fixed a rare race condition in our session h... by m463 on Thursday March 18, 2021 @07:32PM (#26508700)
I have to say the presentation is exceptional, including very good diagrams of the problem.

But there's a thought in the back of my head wondering what kind of stuff hit the fan, and wondering who had to create the presentation. Was it the person who checked in the broken code, was it the bug fixer or his manager, or was it a pm that got stuck having to write up what happened?

Re:We found and fixed a rare race condition in our session h... by fotta on Thursday March 18, 2021 @07:39PM (#26508754)
My guess is that someone close to the remediation wrote it (or it was written by the eng team) and it went through 5 levels of editing before publication.
Re:We found and fixed a rare race condition in our session h... by m463 on Thursday March 18, 2021 @08:00PM (#26508894)
I wonder about the email thread. how many recipients on the TO, how many on the CC (how many on BCC?)
Re:We found and fixed a rare race condition in our session h... by codetrotter on Thursday March 18, 2021 @08:07PM (#26508932)
> how many on BCC?

I guess the only way we will know that is if there’s ever a vulnerability in the session handling code of the mail server that they use also xD

Re:We found and fixed a rare race condition in our session h... by petters on Thursday March 18, 2021 @06:23PM (#26508157)
I agree with you in general, but in this particular case they knowingly deployed thread-unsafe code in production while increasing the threaded concurrency. That should have broken internal development rules.
Re:We found and fixed a rare race condition in our session h... by eli on Thursday March 18, 2021 @06:47PM (#26508348)
Now imagine what happens behind the scenes at other service providers. I’d bet the ones that have policies that bury the details of incidents, on average, have worse problems.
Re:We found and fixed a rare race condition in our session h... by yuliyp on Thursday March 18, 2021 @07:22PM (#26508628)
Yes. Writing code with bugs breaks internal development rules. The issue wasn't thread-unsafe code. It was confusion about object lifecycle. An env object was assumed to no longer be used by the web framework after a request terminated, while in reality it was recycled for future requests as a performance optimization.
Re:We found and fixed a rare race condition in our session h... by simonw on Thursday March 18, 2021 @07:11PM (#26508536)
It sounds to me like they created new internal development rules as a result, which is a smart response to the issue.
Re:We found and fixed a rare race condition in our session h... by Complexicate on Thursday March 18, 2021 @06:00PM (#26508012)
Agreed. However, I smirked a bit when I read your comment, since this is ultimately coming from Microsoft. Let's hope some of this mentality seeps into other Microsoft projects
Re:We found and fixed a rare race condition in our session h... by thitcanh on Thursday March 18, 2021 @06:06PM (#26508041)
GitHub is owned by Microsoft but as far as I know it operates pretty independently. Let’s hope it stays that way.
Re:We found and fixed a rare race condition in our session h... by craftinator on Thursday March 18, 2021 @06:47PM (#26508359)
At this point we're all just waiting for that last E.
Re:We found and fixed a rare race condition in our session h... by johannes1234321 on Thursday March 18, 2021 @07:25PM (#26508657)
What would be the benefit for Microsoft in that case? - They are transforming themselves from a platform company based around windows, it a platform company based around Azure. GitHub is a relevant part there.

This doesn't mean they are only doing good™ with GitHub, but destroying it and open source vendors would be counted to their interests.

Re:We found and fixed a rare race condition in our session h... by arthurcolle on Thursday March 18, 2021 @06:58PM (#26508424)
I don't understand this reference, can you elucidate
Re:We found and fixed a rare race condition in our session h... by scintill76 on Thursday March 18, 2021 @07:01PM (#26508456)
Re:We found and fixed a rare race condition in our session h... by artificial on Thursday March 18, 2021 @07:01PM (#26508452)
Most likely a reference to Embrace, Extend, Extinguish. A strategy Microsoft employed in the past.
Re:We found and fixed a rare race condition in our session h... by arthurcolle on Thursday March 18, 2021 @07:13PM (#26508559)
Ah thanks, yeah. Familiar with the strategy, but didn't get the connection - been a long day over here
Re:We found and fixed a rare race condition in our session h... by m463 on Thursday March 18, 2021 @07:20PM (#26508611)
We should Eulogize
Re:We found and fixed a rare race condition in our session h... by xxqp on Thursday March 18, 2021 @07:16PM (#26508575)
elucidate, extend, extinguish
Re:We found and fixed a rare race condition in our session h... by edoceo on Thursday March 18, 2021 @06:56PM (#26508414)
Or, pro-actively moving to GitLab - either self-hosted or paid and using GH as a simple mirror for discovery.
Re:We found and fixed a rare race condition in our session h... by TedShiller on Thursday March 18, 2021 @06:41PM (#26508313)
This is such a n00b mistake though, the "greater confidence" is quickly canceled out. I end up having less confidence in this company.
Re:We found and fixed a rare race condition in our session h... by SilverRed on Thursday March 18, 2021 @07:43PM (#26508781)
Every bug is a "n00b mistake" when you are looking right at it. There is not a single large project that is not filled with or at least has had trivial bugs with serious repercussions.
Re:We found and fixed a rare race condition in our session h... by sellyme on Friday March 19, 2021 @12:21AM (#26510129)
> Every bug is a "n00b mistake" when you are looking right at it.

Except the ones that are just the universe conspiring against you, like bit flipping in the SM-1800 because of heavily irradiated cattle being transported on a nearby train line.

Re:We found and fixed a rare race condition in our session h... by TedShiller on Friday March 19, 2021 @04:25AM (#26511102)
If you’re a noob then yes every mistake is a noob mistake
Re:We found and fixed a rare race condition in our session h... by ShroudedNight on Thursday March 18, 2021 @07:49PM (#26508831)
I think it's more a statement of the quality and clarity of the postmortem that this problem appear so obvious in hindsight.