Facebook's Top Open Data Problems

Re:Facebook's Top Open Data Problems by bbillings on Tuesday November 04, 2014 @03:49PM (#8558963)

This is the correct answer. Source: I am a FB employee.

[ Parent | on HN ]

Re:Facebook's Top Open Data Problems by beagle3 on Tuesday November 04, 2014 @04:04PM (#8559032)

Thanks.

So, how much does the 4PB of data go down to when it goes into hive? My guess would be something like 200TB.

Is it zipped (or lz4/lzo/zopfli/lzma whatever)? Or is it just "distilled"?

[ Parent | on HN ]

Re:Facebook's Top Open Data Problems by valarauca1 on Tuesday November 04, 2014 @04:54PM (#8559291)

99% of raw data is useless. Just as a rule.

Most of it likely gets tossed into a program to determine if somebody actually needs to do anything, or if something is actually breaking.

For example raw user interaction doesn't really grow. While the event is likely ~1kb or so of raw data, at the end your just incrementing a 64bit counter.

This is a baseless exaggerated post but should shed some light on.

[ Parent | on HN ]

Re:Facebook's Top Open Data Problems by beagle3 on Wednesday November 05, 2014 @07:33AM (#8561487)

> 99% of raw data is useless. Just as a rule.

Well, as a non-facebook user, I think 99.9999% of facebook data is useless :) But facebook is in the habit of tracking everyone's surfing habits across the web (through "like" links), and I assume they do more than just "increment a counter with it", even if they don't keep every single detail.

[ Parent | on HN ]