I think it's impressive that they somehow found 33KB/second worth of data to log for each stream. I can't even imagine the amount of useless shit that must be logged to get to that number.

This is where I'm at. Like thats honestly not much log data. But what are they actually logging? I imagine there is a LOT of repetitive data.

Detailed logging can function as an on-demand APM. Not a bad idea if you have the bandwidth and storage for it.

I think the impressive thing is how much data that is for each user-minute. What could they possibly be storing in 33KB for each second of Netflix you stream?

That's per user. So a million (or ten, 50...) active users means a lot more per minute.

i think you and the above poster are in vehement agreement. ingesting 2 MBs of logs per minute is impressive in its pluperfect unimpressiveness.

maybe the presentation was called "Timmy's first named pipe" or "Sally explores /etc/logrotate.d"

That’s a LOT of text to describe me sitting on my couch. 2MB per minute is far more than the most detailed biography in existence.

220m users. Let’s imagine 50m are streaming concurrently. That’s 100TB an hour in logs lol. They could be storing an entire petabyte of logs a day. My friend did some data center stuff for the large hadron collider and wasn’t hitting these data ingestion states, and these are just to record me binging the office.

The comment said "log processing system". Sounds more like it's a stream and not stored logs.

2 MB/per minute/per stream at Netflix scale is crazy.