Most likely that the 300PB are distilled/normalized/compacted data whereas the 4PB per day are raw logs.

This is the correct answer. Source: I am a FB employee.

Thanks.

So, how much does the 4PB of data go down to when it goes into hive? My guess would be something like 200TB.

Is it zipped (or lz4/lzo/zopfli/lzma whatever)? Or is it just "distilled"?

99% of raw data is useless. Just as a rule.

Most of it likely gets tossed into a program to determine if somebody actually needs to do anything, or if something is actually breaking.

For example raw user interaction doesn't really grow. While the event is likely ~1kb or so of raw data, at the end your just incrementing a 64bit counter.

This is a baseless exaggerated post but should shed some light on.

> 99% of raw data is useless. Just as a rule.

Well, as a non-facebook user, I think 99.9999% of facebook data is useless :) But facebook is in the habit of tracking everyone's surfing habits across the web (through "like" links), and I assume they do more than just "increment a counter with it", even if they don't keep every single detail.

Also 800.000 tables? Surely not in the sense I'm used to, that is, normalized forms where a table corresponds roughy to some business object/noun? Does table mean something else or are there 800k different types of data in there?

Yeah that's hard to wrap my head around. We have 2k in a large app and that's daunting. Really curious why that came to be (though that's probably the least interesting thing in this article)

Data is not stored forever. If most data is only stored ~30 days, then the numbers make sense.