News for hackers. Stuff that matters.
Re:A behind-the-scenes look at Facebook release engineering by benjaminwootton on Thursday April 05, 2012 @12:31PM (#3803330)
Still though, the 1.5 GB binary takes 30 minutes to get out to the servers as per the article.

In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30.

I agree that you can and should have all of the gating and safety nets and checks in the world as you roll forward, but at the end of the day, you need the ultimate safety net of a very fast, very dependable rollback that you can run at will.

IMO reverting isn't for losers. It should be the first thing you do in case of errors, if your architecture supports it.

Re:A behind-the-scenes look at Facebook release engineering by rguldener on Thursday April 05, 2012 @12:47PM (#3803407)
"In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30."

According to Rossi the servers keep the old version around after rollout, so rollback wouldn't require them to push out the old binary again but just to restart the old process. This will probably only take a couple of seconds making rollback really fast if needed.