News for hackers. Stuff that matters.
Re:Reconstructing Twitter's Firehose by ohadron on Thursday February 28, 2019 @02:08AM (#19269569)
To store such a firehose stream of data, you will need approximately 0.3gb of storage per one second of data.

This is if you only collect username, timestamp and tweet, excluding any additional metadata such as data center, likes and retweets, not to mention images and videos.

Full calculation here: https://docs.google.com/spreadsheets/d/1BIAguT9Qvy0GK-dalpQf...

Re:Reconstructing Twitter's Firehose by keithwinstein on Thursday February 28, 2019 @03:58AM (#19270023)
I believe your calculations are too high by a factor of 1000... 53 bytes/(typical tweet) * 6000 typical tweets/second is 318 KB (or 0.3 MB) per second, not 318 MB.
Re:Reconstructing Twitter's Firehose by ohadron on Thursday February 28, 2019 @08:29AM (#19271178)
Yikes, you are totally right. Fixed
Re:Reconstructing Twitter's Firehose by thirit on Thursday February 28, 2019 @02:19AM (#19269612)
Since a couple of years ago the tweet limit was bumped to 280 characters, so I assume the amount is double of what you calculated.
Re:Reconstructing Twitter's Firehose by ohadron on Thursday February 28, 2019 @02:21AM (#19269618)
Good point! Fixing my calculation. So it's between 0.3gb (typical tweet is ~30 chars) and 6.8gb per second