News for hackers. Stuff that matters.
Re:Why are Facebook, Digg and Twitter So Hard To Scale? by dasil003 on Tuesday October 13, 2009 @02:46PM (#879760)
I think it's laughable to put Twitter and Digg in the same category of scalability as Facebook.

Maybe things have changed since the last time I visited Digg over 2 years ago, but the social networking aspects are not very significant. The vast majority of their hits are practically fully page cacheable.

Twitter at least has an interesting scaling problem, but they don't have any features and they move at a glacial pace.

Facebook on the other hand has a graph that almost as nasty as Twitter's (minus the million followers thing), but they have 100 times the features, and they push new code every week.

Re:Why are Facebook, Digg and Twitter So Hard To Scale? by ivankirigin on Tuesday October 13, 2009 @04:26PM (#879998)
There are facebook pages with millions of fans. If you fan a page, you'll "subscribe" its updates too. It is similar to asymmetric following.

The internal representation is quite elegant.

Re:Why are Facebook, Digg and Twitter So Hard To Scale? by yariv on Tuesday October 13, 2009 @11:51PM (#880652)
We (FB) actually push new code every day.
Re:Why are Facebook, Digg and Twitter So Hard To Scale? by timf on Tuesday October 13, 2009 @08:30PM (#880415)
Facebook also "has about 30,000 servers supporting its operations, hosts 80 billion photos, and serves up more than 600,000 photos to its users every second."

(check out http://www.facebook.com/Engineering?v=wall )

Re:Why are Facebook, Digg and Twitter So Hard To Scale? by yarapavan on Wednesday October 14, 2009 @01:29AM (#880767)
Facebook handles 30K machines, 300M active users, 20B photos, & 25TB per day of logging data. http://bit.ly/EA3oY
Re:Why are Facebook, Digg and Twitter So Hard To Scale? by jdrock on Tuesday October 13, 2009 @05:23PM (#880121)
FB should be more difficult as well because they aren't just showing streaming information. They're making (supposedly) intelligent decisions on what information to show you. Twitter is just showing you a "dumb" stream.
Re:Why are Facebook, Digg and Twitter So Hard To Scale? by litewulf on Tuesday October 13, 2009 @05:30PM (#880133)
At the same time, facebook can simply drop messages on the floor if they're bottlenecked, so they have less strict scaling requirements.
Re:Why are Facebook, Digg and Twitter So Hard To Scale? by gaius on Wednesday October 14, 2009 @03:38AM (#880898)
Twitter has guaranteed delivery? Are you sure?

The killer app of Facebook - poking - does need to be reliable ;-)

Re:Why are Facebook, Digg and Twitter So Hard To Scale? by litewulf on Thursday October 15, 2009 @03:41PM (#884102)
I'm pretty sure Twitter is "guaranteed delivery", but thats just my naive user expectation.

Poking, while needing to be reliable is a 1:1 activity, so light on writes and easy to read, whereas broadcasting is difficult because of the wide variance in update frequency and number of followers from user to user.

Re:Why are Facebook, Digg and Twitter So Hard To Scale? by mikeryan on Tuesday October 13, 2009 @05:27PM (#880127)
Just because Twitter and Digg don't operate at the same scale as Facebook doesn't mean they don't face similar issues.

I actually think having a bunch of large sites tackling similar issues will present a whole bunch of neat new technologies for addressing these sorts of issues in the near future.

Re:Why are Facebook, Digg and Twitter So Hard To Scale? by dasil003 on Tuesday October 13, 2009 @05:36PM (#880142)
I'm not talking about absolute scale. Facebook is small compared to Google scale, but in my (probably poorly) informed opinion it's not as impressive.

All these sites have scaled to multiple servers, which means they've all had to address some of the hard problems, and no doubt that many of the issues are very similar. However even if Facebook's traffic were half of Twitter's it would still be a much more impressive architecture. They support a full-on integrated development platform for crying out loud. That is orders of magnitude more impressive than Twitter's lightweight API whether they're serving 100 or 100,000 requests per second.

I'm not dissing Twitter either, it's just that what Facebook has going is pretty incredible. Consider that failing to scale is basically what killed Friendster.

Re:Why are Facebook, Digg and Twitter So Hard To Scale? by mikeryan on Tuesday October 13, 2009 @06:14PM (#880217)
I understand what you're saying Facebook is massively complex compared to digg and twitter - which is likely why it chews up the majority of the article. But I'm not sure why you can't have a top level discussion about scaling (which seems to be the point of the article) that is relevant to all three.
Re:Why are Facebook, Digg and Twitter So Hard To Scale? by dasil003 on Tuesday October 13, 2009 @06:35PM (#880252)
okay fair point.
Re:Why are Facebook, Digg and Twitter So Hard To Scale? by pavs on Tuesday October 13, 2009 @06:52PM (#880276)
"However even if Facebook's traffic were half of Twitter's"

Huh?

Re:Why are Facebook, Digg and Twitter So Hard To Scale? by dasil003 on Tuesday October 13, 2009 @07:03PM (#880293)
"only half"
Re:Why are Facebook, Digg and Twitter So Hard To Scale? by rythie on Tuesday October 13, 2009 @07:28PM (#880330)
Actually I've found that people follow twice as many people on Twitter as on Facebook and receive updates a whole load more often. Facebook does benefit from being a older company with a lot more engineers though.
Re:Why are Facebook, Digg and Twitter So Hard To Scale? by joe_the_user on Tuesday October 13, 2009 @04:08PM (#879971)
What makes Twitter hard is that it is a semi-real-time communications medium that is no longer particularly web-based - a good portion of Tweets come from phones.

But you're right with Digg, I don't see it having any excuses