News for hackers. Stuff that matters.
Re:A behind-the-scenes look at Facebook release engineering by flyt on Thursday April 05, 2012 @12:25PM (#3803294)
Facebook is updating tens of thousands of servers with every push. "Rolling back" a release could take as long as a regular push and contribute to problems as the version in use diverges.

Instead, FB has an aggressive and flexible internal system for "ungating" features to groups based on different criteria. Usually a feature would be pushed out in a deactivated state, then a developer will slowly ramp up its exposure to actual traffic. This limits the ability for a push to insta-break the site and means they can come back around for the next day's push with tweaks, then increase the code's exposure.

Re:A behind-the-scenes look at Facebook release engineering by numlocked on Thursday April 05, 2012 @12:47PM (#3803405)
According to the article reverting does NOT involve re-deploying. Each server maintains the previous version of the binary and if needed the release team can pull the switch to revert all the servers. I assume that takes seconds, not 30 mins.
Re:A behind-the-scenes look at Facebook release engineering by TeeWEE on Thursday April 05, 2012 @05:17PM (#3804680)
In fact i worked at a dutch social network where we also used hiphop. The new compiled binary is pushed to all servers and then it is started, the old binary is stopped and a port handover is done. Thus deploying without downtime. The old binary is available on the system, making a rollback very fast. However old binaries are removed after a time. So you can only roll back to a previous version quickly.
Re:A behind-the-scenes look at Facebook release engineering by ironchef on Thursday April 05, 2012 @12:49PM (#3803418)
"Facebook is updating tens of thousands of servers with every push. "Rolling back" a release would effectively take as long as a regular push and could contribute to problems if they're found." Absolutely not. A lot of places get around this with a simple symlink switch. You keep N older releases. So it might be /code/releases with various datestamped releases...and /code/releases/current points to the currently running one. Want to rollback? Point current to one older. Done.
Re:A behind-the-scenes look at Facebook release engineering by MatthewPhillips on Thursday April 05, 2012 @01:08PM (#3803493)
Presumably the 1.5gb facebook binary is deployed as a daemon listening on port 80. Seems unlikely that they are cgi-ing a new process for each request.
Re:A behind-the-scenes look at Facebook release engineering by rhizome on Thursday April 05, 2012 @02:20PM (#3803843)
CGI?
Re:A behind-the-scenes look at Facebook release engineering by darklajid on Thursday April 05, 2012 @03:47PM (#3804268)
I'm torn. I understand your question, but yeah - what is that binary doing?

It is compiled PHP. Do they embed an http server? Do they talk fcgi with a well-known webserver, like - php would?

Re:A behind-the-scenes look at Facebook release engineering by HarrisonFisk on Thursday April 05, 2012 @04:42PM (#3804524)
HPHP has a built in multi-threaded webserver.
Re:A behind-the-scenes look at Facebook release engineering by benjaminwootton on Thursday April 05, 2012 @12:31PM (#3803330)
Still though, the 1.5 GB binary takes 30 minutes to get out to the servers as per the article.

In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30.

I agree that you can and should have all of the gating and safety nets and checks in the world as you roll forward, but at the end of the day, you need the ultimate safety net of a very fast, very dependable rollback that you can run at will.

IMO reverting isn't for losers. It should be the first thing you do in case of errors, if your architecture supports it.

Re:A behind-the-scenes look at Facebook release engineering by rguldener on Thursday April 05, 2012 @12:47PM (#3803407)
"In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30."

According to Rossi the servers keep the old version around after rollout, so rollback wouldn't require them to push out the old binary again but just to restart the old process. This will probably only take a couple of seconds making rollback really fast if needed.