- the data we used was crawled by Kwak et. al in 2009. We wanted to use a real social network dataset for the experiment and that was the largest/most useful one we could find. Other than de-duplication we did not make any modifications to the dataset, so the statistics reported in their paper still hold: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.153....
- You mean what is the overhead induced by pre-computing the stream edge rather than collecting the relevant streams at query time? You are right that this requires a significant amount of storage space, however, as you also pointed out, this will get cold quickly and be sitting on disk only (i.e. not taking up space in the valuable cache). The reason this is very efficient is because of the time-based vertex centric index we build for the stream edges. This allows us to quickly pull out the most recent tweets for any user. If we had to compute those at query time, we would have to traverse to each person followed, get their 10 most recent tweets and then merge those in-memory. That would be significantly more expensive and since stream reading is probably the most frequent activity on twitter, pre-computing it saves a lot of time at the expense of inexpensive disk storage.
I'd like to see a few other details that aren't mentioned:
- What's the following distribution end up looking like? Does it have a similar fraction of 'celebrity' users with huge follower counts? Or more technically, does the russian roulette against the recommendation sampling end up producing a network similar to a scale free graph grown via preferential attachment (Barabási–Albert model)? It looks like your mean fanout is about 20, which is smaller than what twitter has published, but I'd be more interested in knowing how many 10k+ follower users are in the graph.
- What's the write amplification like? ~1.6 Billion per tweet per follower edges stored daily seems like it could burn a lot of capacity quickly, though most of it will grow cold quickly and could be pushed to archive. Making a rough guess from your disk write monitoring line graph, it looks like you'd be putting down about 16GB a day? It'd be interesting to see a comparison between this run and one done where streams are built indirectly via follower links alone.