Re:Reconstructing Twitter's Firehose
by stuck_in_matrix on Wednesday February 27, 2019 @09:10PM
(#19268496)
When I did the analysis, I was puzzled why certain machines handle a higher percentage of tweets compared to others -- so you are most likely correct that there may be some geographic consideration to the distribution.
I'm rewriting the code to include a prescan of the time range to determine which server ids are in play at the time and which server ids are most active.
Figuring out how to deconstruct Snowflake was challenging and there is still a lot of analysis left to do.
Re:Reconstructing Twitter's Firehose
by detaro on Thursday February 28, 2019 @04:20AM
(#19270100)
How do you read the code of an implementation detail of Twitter's servers? There's no guarantee that the example code they released years ago still matches what they use.
I'm rewriting the code to include a prescan of the time range to determine which server ids are in play at the time and which server ids are most active.
Figuring out how to deconstruct Snowflake was challenging and there is still a lot of analysis left to do.