News for hackers. Stuff that matters.
Re:How Facebook Ships Code by gouranga on Thursday July 05, 2012 @05:46AM (#4202380)
What a crock of shit:

after boot camp, all engineers get access to live DB

I can understand on a startup or small org but an organisation of that size, there should be very tight access control.

Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else.

I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be.

I'd never put my data near FB. They are simply irresponsible.

Re:How Facebook Ships Code by piggity on Thursday July 05, 2012 @07:01AM (#4202531)
You finance and billing guys will never understand.

These are rockstars.

They work for facebook.

They would never ever type UPDATE users SET email = username || '@facebook.com'; WHERE username == 'john.smith';

Re:How Facebook Ships Code by jacques_chester on Thursday July 05, 2012 @07:14AM (#4202576)
Of course not. Much faster to type

    UPDATE users SET email = username || '@facebook.com';
Then let the users sort out the exceptions.
Re:How Facebook Ships Code by alttab on Thursday July 05, 2012 @08:34AM (#4202738)
I think he was being sarcastic - you can see the semicolon in the middle of the command which looks like pretty much what they did with the whole e-mail debacle. You simply removed the second incomplete statement.
Re:How Facebook Ships Code by jacques_chester on Thursday July 05, 2012 @08:37AM (#4202744)
I now realise that I am underqualified to work at Facebook. :(
Re:How Facebook Ships Code by getsat on Thursday July 05, 2012 @11:42AM (#4203504)
>implying this was an engineering team decision
Re:How Facebook Ships Code by gaius on Thursday July 05, 2012 @09:09AM (#4202839)
Right. FB doesn't get that all they have is the user experience, and it is bad user experience if I drop out of a conversation I'm having because their DBA-free database has decided to simply, silently drop my last comment. Or if it takes me several attempts to post a photo. Or if I invite someone to an event, and they never get the invite.

You can do all sorts of amazing things if you just don't care if your code actually works or not. Grown-up companies have full-time professional DBAs, and not just for separation of concerns.

Re:How Facebook Ships Code by darkarmani on Thursday July 05, 2012 @12:07PM (#4203588)
> Grown-up companies have full-time professional DBAs

I don't know if I'd call them "professional," but they definitely have full-time DBAs. ;)

Re:How Facebook Ships Code by mkjones on Friday July 06, 2012 @01:43AM (#4206495)
What do you mean? I've worked with some of our DBAs, and they're quite good. In fact, I can't think of a single site issue that was caused by a DBA. I work on fighting spam at FB, and we make use of mysql quite a bit.
Re:How Facebook Ships Code by jorgeleo on Thursday July 05, 2012 @07:36AM (#4202629)
But Facebook is not a financial institution, and I would put the responsability in the user if financial data is on the news feed.

One size (of engineering practices) fit all mentality is not the correct approach either.

Re:How Facebook Ships Code by gouranga on Thursday July 05, 2012 @10:03AM (#4203035)
You are correct - it is not a financial organsiation.

It does know who you are, what you eat, where you've been, who you're friends with, what you're interested in, what your political allegiances are, what your bowel movements are like etc.

A list of investments or your mortgage statement is way less important.

Re:How Facebook Ships Code by jdsemrau on Thursday July 05, 2012 @07:06AM (#4202547)
>I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be.

Funny, I had the same argument with a colleague today. Since FB is listed and their business model is based on customer data they should follow the same legal requirements as customer finance companies.

Re:How Facebook Ships Code by yashchandra on Thursday July 05, 2012 @08:01AM (#4202678)
"they should follow the same legal requirements"

Perhaps. But they certainly do not have the restrictions of financial audit requirements and things like Dodd Frank etc. which is overwhelming all the major banks right now and is a big way to earn money for consultants if they know ABC of audit/compliance in 2012. Anyway, the point is that fb even though not ideal in how they store/use user data (which I personally am not a fan of as well), it does not matter as much as it matters to a bank/financial institution. I say this while I work for clients that are major banks.

Re:How Facebook Ships Code by jschuur on Thursday July 05, 2012 @06:12AM (#4202436)
Note that it doesn't say whether they all have write access, or whether all the data they have access to is encrypted in some way.

And any of those statements should be interpreted as one response he got from talking to lots of different people, and be subject to some amount of skepticism.

Re:How Facebook Ships Code by gouranga on Thursday July 05, 2012 @07:13AM (#4202572)
Either way, it simply doesn't matter.
Re:How Facebook Ships Code by TazeTSchnitzel on Thursday July 05, 2012 @06:07AM (#4202422)
It's a little irresponsible, but how else do you expect them to debug live code?
Re:How Facebook Ships Code by arethuza on Thursday July 05, 2012 @06:38AM (#4202478)
"how else do you expect them to debug live code"

I've actually spent quite a long time writing code where, for various legal reasons, you are never going to get direct access to the production data. (And indeed some cases where accessing the data in question would be a serious criminal offence).

Makes you really keen on defensive programming and comprehensive logging - although you do then have to be careful about what you log.

I actally found it kind of fun to try and debug complex systems based only on a log file and QA with production operations folks.

[NB Of course, none of this would be applicable to Facebook]

Re:How Facebook Ships Code by gouranga on Thursday July 05, 2012 @07:12AM (#4202564)
Thou who uses unit tests, UI tests, precondition, postcondition checks, heavy logging, knows the language and toolchain and knows arse from elbow NEVER debugs in production out of necessity or desire.

It just works.

In the last 5 years, we've never hooked a debugger to production instance or given access to a developer.

In that time, we've pushed approximately 912 billion SQL transactions through the system and 24 billion page hits (most of our stuff is backend non OLTP).

That's how it's done.

Re:How Facebook Ships Code by rimantas on Thursday July 05, 2012 @07:16AM (#4202579)
No please tell me, how did you learn to write unit test which cover all the crazy data configurations users are able to come with.
Re:How Facebook Ships Code by gioele on Thursday July 05, 2012 @08:38AM (#4202750)
Boundary value analysis? Machine state transitions testing? Equivalence partitioning?

There are heaps of techniques one can use to identify which data points and combinations are useful to test. Some of these techniques even take a peek at your code to highlight the possible pain points.

Re:How Facebook Ships Code by gaius on Thursday July 05, 2012 @09:13AM (#4202859)
LOL! You think someone who's drunk the unit testing kool-aid has even heard of any of that?
Re:How Facebook Ships Code by wpietri on Thursday July 05, 2012 @09:49AM (#4202984)
You have to start somewhere. If I have to choose between a world where people never test and one where they test but aren't very good at it yet, I'll definitely take the latter.

Of course, I'm still filled with rage when I find a cargo-cult suite of unit tests. But it's so much easier to convince those people to write smarter tests than it is to deal with the team that has no time to write tests because they're spending all their time debugging.

Re:How Facebook Ships Code by alttab on Thursday July 05, 2012 @08:36AM (#4202743)
By being able to design software that doesn't need or allow crazy data configurations.

My whole mantra is don't manage complexity: avoid it.

Re:How Facebook Ships Code by gouranga on Thursday July 05, 2012 @09:59AM (#4203017)
You nailed it - thanks for the absolutely spot on comment.

Design is the key word here. It requires thought, it requires intelligence and it requires multiple people's input.

Re:How Facebook Ships Code by TazeTSchnitzel on Thursday July 05, 2012 @07:15AM (#4202577)
You're right.

Now that I think about it, I doubt Facebook is that careful, considering how often they break things.

Re:How Facebook Ships Code by bnr on Thursday July 05, 2012 @06:20AM (#4202451)
You don't. Reproduce the issue on a development instance.
Re:How Facebook Ships Code by damncabbage on Thursday July 05, 2012 @06:25AM (#4202457)
I first took TazeTSchnitzel's comment to be sarcasm, but now I'm not so sure.

(Poe's Law says hi.)

Re:How Facebook Ships Code by TazeTSchnitzel on Thursday July 05, 2012 @06:41AM (#4202484)
Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.
Re:How Facebook Ships Code by mgkimsal on Thursday July 05, 2012 @07:42AM (#4202640)
A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc).

Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports).

Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L.

Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops.

Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.

Re:How Facebook Ships Code by wpietri on Thursday July 05, 2012 @09:59AM (#4203020)
I agree totally with your broad point, but wanted to respond to the "I can't believe a user typed..."

Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume.

Madness, sure, but common madness.

Re:How Facebook Ships Code by tedunangst on Thursday July 05, 2012 @12:42PM (#4203735)
Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops.

Makes me wonder what dynamic language you're used to. An uncaught exception in both python and ruby aborts the program, not just the tiny inner loop that happens to be running.

Re:How Facebook Ships Code by mgkimsal on Thursday July 05, 2012 @05:39PM (#4205119)
Primarily PHP, although I seem to remember classic ASP VBScript not dying if you passed a bad email to CDO (but that's been a long time, so i might not remember that correctly).

PHP would not throw an exception if the mail function got a bad email address. well, using some mailing libraries might, but the core mailing function (basically just a wrapper for sendmail/postfix CLI stuff) wouldn't.

Re:How Facebook Ships Code by tedunangst on Thursday July 05, 2012 @06:11PM (#4205264)
Oh, you expected silent failure from a crappy library. :) I think Java (or rather, the library you're now using) made a much better decision to not blunder along.
Re:How Facebook Ships Code by damncabbage on Thursday July 05, 2012 @09:37PM (#4205959)
"on error resume next" redux?
Re:How Facebook Ships Code by gouranga on Thursday July 05, 2012 @10:05AM (#4203047)
> Adding on top of that was my own dynamic language background not mapping well to the JVM

There's your problem...

Re:How Facebook Ships Code by mgkimsal on Thursday July 05, 2012 @10:52AM (#4203274)
It's a problem, but it certainly wasn't my only - groovy, to be precise.
Re:How Facebook Ships Code by rimantas on Thursday July 05, 2012 @07:18AM (#4202586)
How do you do that, if issue is caused by some corner case with user data you did not foresee and hence not covered?
Re:How Facebook Ships Code by Goladus on Thursday July 05, 2012 @08:51AM (#4202789)
For some applications, the number of corner cases that can not be resolved by reading log files is minimal. (you are writing logs, right?) That is, too, assuming the problem is actually in the code itself and not the infrastructure.