News for hackers. Stuff that matters.
Re:Rainbird: The Way Twitter Counts Tweets In Realtime (Soon... by jhugg on Friday February 04, 2011 @04:20PM (#2181285)
Shameless self-promotion, but important point nonetheless:

VoltDB is truly fantastic at this kind of workload. It's one of the first use cases we had traction with. Also, VoltDB is open source today.

Re:Rainbird: The Way Twitter Counts Tweets In Realtime (Soon... by lsb on Saturday February 05, 2011 @09:30AM (#2183127)
VoltDB looks pretty awesome, but I'm pretty concerned by its lack of ability to join a table to itself, to have over six tables in a join, to aggregate or select distinct arbitrary values. http://community.voltdb.com/docs/ReleaseNotes/index
Re:Rainbird: The Way Twitter Counts Tweets In Realtime (Soon... by jhugg on Saturday February 05, 2011 @12:40PM (#2183490)
At VoltDB, we're working on improving our SQL support. Initially, we focused on core SQL useful for OLTP. We've added functionality with every release and we plan to continue in 2011.

For example, the March release of VoltDB will support more than 6 table joins and has some improvements to aggregation and distinct code. It also has a much more usable explain plan feature.

We feel that VoltDB offers one of the richer query interfaces of systems that scale to its level, but we don't plan to sit still.

Re:Rainbird: The Way Twitter Counts Tweets In Realtime (Soon... by charrington on Saturday February 05, 2011 @12:07AM (#2182500)
I think that at the rates Twitter is writing counter data (Many TBs per day denormalized, ~0.5TB normalized), a RAM-based solution like VoltDB would be prohibitively expensive. Rainbird allows Twitter to use cheap disk-based storage but still get acceptable (sub-second) latency.
Re:Rainbird: The Way Twitter Counts Tweets In Realtime (Soon... by jhugg on Saturday February 05, 2011 @12:24AM (#2182523)
Perhaps, but it really depends on what you want to do exactly. You can probably store a lot less and get the same throughput.

- VoltDB isn't log-structured, so you really only have to store the state. How fast you can mutate it isn't limited by RAM amounts. We see use cases with utter firehoses of data that update just tens or hundreds of gigabytes of state.

- Beyond normalization, you can probably reduce the number of redundant counters, e.g. use SQL to count which URLs start with "amazon". This would be painful in many systems, but depending on the query, can often be done at scale in VoltDB.

- The byte overhead per counter is also likely much lower in an ACID/Relational store.

Finally, VoltDB is designed to migrate data to disk based stores (such as Hadoop or an OLAP store) as memory fills up. This is a feature we're working very hard on see as a big differentiator. It adds complexity if you need to query across stores, but you get a best-of-both-worlds feature set.

Re:Rainbird: The Way Twitter Counts Tweets In Realtime (Soon... by JoachimSchipper on Saturday February 05, 2011 @06:16AM (#2182904)
I didn't know VoltDB was open source, actually. Cool!

Is the database really entirely in-memory, though?

Re:Rainbird: The Way Twitter Counts Tweets In Realtime (Soon... by jhugg on Saturday February 05, 2011 @12:57PM (#2183528)
VoltDB relies on syncronous replication to protect data from hardware and network failures.

It also supports continuous and transactionally-isolated snapshotting to disk.