Facebook's e-mail debacle: One 'bug' fix, but rollback impossible

Re:Facebook's e-mail debacle: One 'bug' fix, but rollback im... by ChuckMcM on Monday July 02, 2012 @10:40PM (#4192230)

Ouch. Talk about not thinking it through. Relating back to a common topic here, one of the things that distinguishes between senior engineers and not senior engineers in my experience is the ability to visualize both secondary and tertiary consequences of a particular change to the system. They stop and say "Ok, we change this, that means ..." and then walk forward past the change in their mind exploring the ramifications. In order to do that they have to have a broad understanding of the system involved from both a technical and operational perspective. They come back with "Oh if we do that then we'll get this effect and that would be bad, how can we prevent that from happening?" and if the answer was "We can't." then they spike the feature.

One of the most frustrating things for me as an engineer at Sun early on was bouncing some whiz bang idea off Bill Joy only to get one of his one line zinger 'but that breaks X' type replies. Always annoying but often right as he would take what was proposed, extrapolate two, three, or a half dozen steps and then point out the now 'obvious' flaw.

[ Parent | on HN ]

Re:Facebook's e-mail debacle: One 'bug' fix, but rollback im... by jmspring on Tuesday July 03, 2012 @12:55AM (#4192596)

Facebook certainly has a reputation in engineering circles as following the "move fast and break things" model or touting "eventual consistency" or similar. That works for certain areas, as others have pointed out, but once you start manipulating customer data that they have set up, it crosses a line.

I'm a strong believer in confining Windows to a fixed set of resources -- i.e., a VM with fixed memory and disk space. This incident, though I don't seem to be directly impacted (checked contacts, account settings, etc), Facebook has crossed that line into Windows equivalency for me. It needs to be confined where possible.

Mobile app deleted on all my iOS devices and FB account disassociated on my WP7 device (most used for testing).

[ Parent | on HN ]

Re:Facebook's e-mail debacle: One 'bug' fix, but rollback im... by codelust on Tuesday July 03, 2012 @01:32AM (#4192689)

This.

The problem is not limited to engineering. I have often had to push back with product guys responding to feature requests from users to address edge cases.

The trouble with doing so in social networks is that the potential for the law of untended consequences to strike is much greater in it as it is nearly impossible to test for every permutation and combination available.

[ Parent | on HN ]

Re:Facebook's e-mail debacle: One 'bug' fix, but rollback im... by rhizome on Tuesday July 03, 2012 @01:07PM (#4195018)

If, as described elsewhere, the overwriting is due to using a specific and common API, the only unintended consequence is user reaction. No need to test for every possibility there.

[ Parent | on HN ]

Re:Facebook's e-mail debacle: One 'bug' fix, but rollback im... by rhubarbquid on Monday July 02, 2012 @11:42PM (#4192399)

"move fast and break things"

[ Parent | on HN ]

Re:Facebook's e-mail debacle: One 'bug' fix, but rollback im... by Schwolop on Tuesday July 03, 2012 @12:15AM (#4192475)

Let's hope that changes to "hurry up and fix things".

[ Parent | on HN ]

Re:Facebook's e-mail debacle: One 'bug' fix, but rollback im... by markokocic on Tuesday July 03, 2012 @05:57AM (#4193318)

Please don't. If they really hurry up to fix things they could just create another breakage ;) Let them time to really think through it and fix the damage, instead of trying to do a "quick fix in production".

[ Parent | on HN ]

Re:Facebook's e-mail debacle: One 'bug' fix, but rollback im... by SimHacker on Tuesday July 03, 2012 @10:52AM (#4194310)

If only somebody's bounced the idea of Solaris off of him... ;(

[ Parent | on HN ]

Re:Facebook's e-mail debacle: One 'bug' fix, but rollback im... by ChuckMcM on Tuesday July 03, 2012 @12:44PM (#4194890)

Sadly Solaris, aka the 'Lulu' project, aka the merging of SunOS and System V, was in fact the brain child of Eric Schmidt and Bill Joy. They stood on the podium and announced it together in October 1989. Folks would have noticed if the stock market hadn't gone into free fall :-)

[ Parent | on HN ]