News for hackers. Stuff that matters.
Re:Reconstructing Twitter's Firehose by aboutruby on Wednesday February 27, 2019 @06:06PM (#19267713)
That's quite amazing!

One thing is that the data center ids are in the tweet ids, so it could be used to get a rough location of Twitter users.

Re:Reconstructing Twitter's Firehose by aboutruby on Wednesday February 27, 2019 @08:28PM (#19268325)
I see there is interest in this observation so I used a little ruby (.to_s(2)[-22..-17].to_i(2)) to get the datacenter id.

Then ran it on a few Twitter accounts: https://pastebin.com/w8Dnj5kM

It does work, and it's going to be hard to patch

edit: I realized you don't only get one location but the whole location history of a Twitter user. Also locating Twitter's data centers as it doesn't seem to be public information

Re:Reconstructing Twitter's Firehose by stuck_in_matrix on Wednesday February 27, 2019 @08:50PM (#19268413)
This is really interesting. When I did the original analysis on datacenter / server ids, I didn't think about correlation with user accounts. Nice observation!
Re:Reconstructing Twitter's Firehose by aboutruby on Wednesday February 27, 2019 @11:17PM (#19268915)
I was curious and just did that :) https://gist.github.com/localhostdotdev/48ed13972c3e5391a47f... (small sample of ~1500 localized tweets)
Re:Reconstructing Twitter's Firehose by Sommer on Wednesday February 27, 2019 @09:52PM (#19268653)
Should be pretty simple to check the tweet geo or location mentions per server id to see of they imply a correlation with geographic area of the server. Then you're just one hop to knowing where (or where not) other tweeters are.
Re:Reconstructing Twitter's Firehose by jaytaylor on Wednesday February 27, 2019 @11:56PM (#19269032)
They could "fix" it by periodically rotating the DC and server IDs.

I wish it was not so easily fixable, because this will break the key space reduction trick ;), but unfortunately such a solution is feasible and would come with the side-effect of drastically increasing the required scanning space.

Re:Reconstructing Twitter's Firehose by atian on Thursday February 28, 2019 @01:31AM (#19269415)
What are you guys even smoking. The ID segment is 5 bits long. You have an extra server ID bit making the datacenter results more significant than they are.