Twitter to Release All Tweets to Scientists

Re:Twitter to Release All Tweets to Scientists by StavrosK on Tuesday May 27, 2014 @06:34AM (#7803754)

Jeez, 400 GB of text per day? How the hell?

EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?

[ Parent | on HN ]

Re:Twitter to Release All Tweets to Scientists by nevinera on Tuesday May 27, 2014 @07:47AM (#7803931)

Most of that data is not the content of the tweet itself, but the metadata associated with it. When I last checked, we were storing about a kilobyte of data for every tweet.

Also, tweets are limited to 140 characters, not bytes - chinese tweets typically take about 200-250 bytes, for example.

[ Parent | on HN ]

Re:Twitter to Release All Tweets to Scientists by StavrosK on Wednesday May 28, 2014 @10:47PM (#7814435)

Yeah, I figured 300 bytes per Tweet, to be generous, but didn't realize it would take 1 kb of metadata. Thanks for that detail.

[ Parent | on HN ]

Re:Twitter to Release All Tweets to Scientists by JamesMcMinn on Tuesday May 27, 2014 @09:52AM (#7804421)

I have access to the Twitter gardenhose (which is equal to slightly less than 10% of the full volume). These are the RX and TX statistics from the machine that I've been using to gather data for a few months now:

RX bytes:18053724505080 (16.4 TiB) TX bytes:2686623557042 (2.4 TiB)

It works out at around 70GB/day, so I'd actually think that the full firehose would use considerably more data than 400GB/day (likely closer to 800GB).

[ Parent | on HN ]

Re:Twitter to Release All Tweets to Scientists by vschiavoni on Tuesday May 27, 2014 @06:57AM (#7803822)

Have you ever heard about the 'Human Brain Project' ?

Well, they produce 100 GB of data _per second_

Welcome to the real big data world.

[ Parent | on HN ]

Re:Twitter to Release All Tweets to Scientists by vijayr on Tuesday May 27, 2014 @11:36AM (#7804986)

Woah, 100 GB per second???!! What exactly do they do?

Edit: Never mind, this is what they are doing

"We would like to develop some kind of ‘google' brain where we can zoom in and out, see it from different perspectives and understand how brain structure and function is related."

[ Parent | on HN ]

Re:Twitter to Release All Tweets to Scientists by Wilya on Tuesday May 27, 2014 @07:45AM (#7803926)

Depends how much metadata there is with every tweet.

I took a random tweet on my timeline, and the JSON representation, with added metadata, weighs 8740 bytes.

With this figure, you would "only" need 500 tweets/sec to get 400GB a day.

There's a lot of people on Twitter, and a lot of bots, so that doesn't seem unreasonable to me.

[ Parent | on HN ]

Re:Twitter to Release All Tweets to Scientists by maxerickson on Tuesday May 27, 2014 @07:31AM (#7803890)

Pick a topic you are familiar with. Open up a twitter search for it. Wait a while. See the inevitable storm of tweet-spam that sort of looks like social sharing.

Thousands of fake accounts are tweeting out nonsense all the time. Another example, there are multiple accounts that tweet items from HN (and presumably lots of other rss feeds).

[ Parent | on HN ]

Re:Twitter to Release All Tweets to Scientists by Houshalter on Tuesday May 27, 2014 @07:05AM (#7803829)

Maybe it includes images.

[ Parent | on HN ]

Re:Twitter to Release All Tweets to Scientists by cmyr on Tuesday May 27, 2014 @09:58AM (#7804466)

it does not include images. A tweet object does include a bunch of metadata though: https://dev.twitter.com/docs/platform-objects/tweets

[ Parent | on HN ]