News for hackers. Stuff that matters.
Re:How Facebook Ships Code by bnr on Thursday July 05, 2012 @06:20AM (#4202451)
You don't. Reproduce the issue on a development instance.
Re:How Facebook Ships Code by damncabbage on Thursday July 05, 2012 @06:25AM (#4202457)
I first took TazeTSchnitzel's comment to be sarcasm, but now I'm not so sure.

(Poe's Law says hi.)

Re:How Facebook Ships Code by TazeTSchnitzel on Thursday July 05, 2012 @06:41AM (#4202484)
Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.
Re:How Facebook Ships Code by mgkimsal on Thursday July 05, 2012 @07:42AM (#4202640)
A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc).

Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports).

Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L.

Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops.

Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.

Re:How Facebook Ships Code by wpietri on Thursday July 05, 2012 @09:59AM (#4203020)
I agree totally with your broad point, but wanted to respond to the "I can't believe a user typed..."

Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume.

Madness, sure, but common madness.

Re:How Facebook Ships Code by tedunangst on Thursday July 05, 2012 @12:42PM (#4203735)
Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops.

Makes me wonder what dynamic language you're used to. An uncaught exception in both python and ruby aborts the program, not just the tiny inner loop that happens to be running.

Re:How Facebook Ships Code by mgkimsal on Thursday July 05, 2012 @05:39PM (#4205119)
Primarily PHP, although I seem to remember classic ASP VBScript not dying if you passed a bad email to CDO (but that's been a long time, so i might not remember that correctly).

PHP would not throw an exception if the mail function got a bad email address. well, using some mailing libraries might, but the core mailing function (basically just a wrapper for sendmail/postfix CLI stuff) wouldn't.

Re:How Facebook Ships Code by tedunangst on Thursday July 05, 2012 @06:11PM (#4205264)
Oh, you expected silent failure from a crappy library. :) I think Java (or rather, the library you're now using) made a much better decision to not blunder along.
Re:How Facebook Ships Code by damncabbage on Thursday July 05, 2012 @09:37PM (#4205959)
"on error resume next" redux?
Re:How Facebook Ships Code by gouranga on Thursday July 05, 2012 @10:05AM (#4203047)
> Adding on top of that was my own dynamic language background not mapping well to the JVM

There's your problem...

Re:How Facebook Ships Code by mgkimsal on Thursday July 05, 2012 @10:52AM (#4203274)
It's a problem, but it certainly wasn't my only - groovy, to be precise.
Re:How Facebook Ships Code by rimantas on Thursday July 05, 2012 @07:18AM (#4202586)
How do you do that, if issue is caused by some corner case with user data you did not foresee and hence not covered?
Re:How Facebook Ships Code by Goladus on Thursday July 05, 2012 @08:51AM (#4202789)
For some applications, the number of corner cases that can not be resolved by reading log files is minimal. (you are writing logs, right?) That is, too, assuming the problem is actually in the code itself and not the infrastructure.