News for hackers. Stuff that matters.
Re:Manhattan: Real-time, multi-tenant distributed database f... by jandrewrogers on Wednesday April 02, 2014 @02:38PM (#7517921)
A single tweet is closer to 2500 bytes on average in most of the data feeds. However, your point still remains that this a "human-scale" data model that is relatively small and with a low data rate. It is pretty easy to engineer systems that can keep up with this even if you are not an expert in designing real-time database systems.

By comparison, many complex machine-generated data sources (e.g. real-time entity tracking) that are sometimes fused with the Twitter firehose operate at millions of complex records every second (often tens of gigabytes per second) that need to be processed, indexed, and analyzed in real-time. You can't deal with this kind of data model using something like Twitter's current architecture because the several order of magnitude difference in velocity and volume exposes the limitations of most database platform designs people typically use.

There are no comments on this comment yet.