stuck_in_matrix
stuck_in_matrix on Hacker News
Re:Reconstructing Twitter's Firehose
on Wednesday February 27, 2019 @09:17PM (#19268521)
Python is perfectly fine for these type of tasks. I ingest all of Reddit in real-time (https://pushshift.io) and also ingest Gab.com and several others (Stackoverflow, etc.) and at most one or two CPU cores are at 10-15%.
Also, when I provide code examples, I try and use a language that most programmers will have some exposure to and generally Python is high on that list.
Re:Reconstructing Twitter's Firehose
on Wednesday February 27, 2019 @09:10PM (#19268496)
When I did the analysis, I was puzzled why certain machines handle a higher percentage of tweets compared to others -- so you are most likely correct that there may be some geographic consideration to the distribution.
I'm rewriting the code to include a prescan of the time range to determine which server ids are in play at the time and which server ids are most active.
Figuring out how to deconstruct Snowflake was challenging and there is still a lot of analysis left to do.
Re:Reconstructing Twitter's Firehose
on Wednesday February 27, 2019 @09:05PM (#19268477)
It's variable based on what's going on at the time but I've seen upwards of 7k tweets a second for the sections of the timeline that I've ingested using this technique.
Someone suggested trying it when the New Year starts in Japan. Apparently there are tens of thousands of tweets per second then.
Re:Reconstructing Twitter's Firehose
on Wednesday February 27, 2019 @08:50PM (#19268413)
This is really interesting. When I did the original analysis on datacenter / server ids, I didn't think about correlation with user accounts. Nice observation!
Re:Reconstructing Twitter's Firehose
on Wednesday February 27, 2019 @08:48PM (#19268409)
I am the author of this document. If anyone has any questions, I'd be happy to answer them!