Re:Reconstructing Twitter's Firehose
by aboutruby on Wednesday February 27, 2019 @08:28PM
(#19268325)
I see there is interest in this observation so I used a little ruby (.to_s(2)[-22..-17].to_i(2)) to get the datacenter id.
Then ran it on a few Twitter accounts: https://pastebin.com/w8Dnj5kM
It does work, and it's going to be hard to patch
edit: I realized you don't only get one location but the whole location history of a Twitter user. Also locating Twitter's data centers as it doesn't seem to be public information
Re:Reconstructing Twitter's Firehose
by stuck_in_matrix on Wednesday February 27, 2019 @08:50PM
(#19268413)
This is really interesting. When I did the original analysis on datacenter / server ids, I didn't think about correlation with user accounts. Nice observation!
Re:Reconstructing Twitter's Firehose
by aboutruby on Wednesday February 27, 2019 @11:17PM
(#19268915)
I was curious and just did that :) https://gist.github.com/localhostdotdev/48ed13972c3e5391a47f... (small sample of ~1500 localized tweets)
Should be pretty simple to check the tweet geo or location mentions per server id to see of they imply a correlation with geographic area of the server. Then you're just one hop to knowing where (or where not) other tweeters are.
Re:Reconstructing Twitter's Firehose
by jaytaylor on Wednesday February 27, 2019 @11:56PM
(#19269032)
They could "fix" it by periodically rotating the DC and server IDs.
I wish it was not so easily fixable, because this will break the key space reduction trick ;), but unfortunately such a solution is feasible and would come with the side-effect of drastically increasing the required scanning space.
One thing is that the data center ids are in the tweet ids, so it could be used to get a rough location of Twitter users.