News for hackers. Stuff that matters.
Re:Ask HN: Cassandra, HBase, Hypertable, Voldemort. Lowest r... by helwr on Tuesday June 29, 2010 @02:47AM (#1470647)
you need to think in terms of reads/sec vs latency, not just latency

in other words 1) what is the throughput (load) you expect your data store to sustain in reads/sec

2) what is the acceptable latency in msec for your application under expected load (see 1)

and 3) how many simultaneous connections your db has to support

to figure out which data store suits you best in terms of capacity/performance you have two choices:

1) run a benchmark on your hardware, ideally with your own client

2) run a benchmark on your hardware, ideally with your own client

i'd recommend starting with this: http://wiki.github.com/brianfrankcooper/YCSB/

dont forget that the tools you mentioned are intended for storing terabytes of data on hundreds/thousands of machines and their key benefit is scaling the bulk throughput rather than optimizing performance

check out MongoDB, Redis, Tokyo Tyrant if you have up to 10-20GB, these are superfast tools but limited to storage/IO/memory capacity of a single box

also look at some of these benchmarks (but running your own test is always better)

http://research.yahoo.com/files/ycsb.pdf

http://www.slideshare.net/schubertzhang/hbase-0200-performan...

http://code.google.com/p/redis/wiki/Benchmarks

http://capttofu.livejournal.com/23381.html

http://colinhowe.wordpress.com/2009/04/27/redis-vs-mysql/

Re:Ask HN: Cassandra, HBase, Hypertable, Voldemort. Lowest r... by ghotli on Tuesday June 29, 2010 @09:26AM (#1471289)
I've benchmarked Tokyo Tyrant, MongoDB, and Redis for a smaller dataset (~20gb). We ended up going with Tyrant because it stored our serialized data more efficiently than the others. That index lives in memory on a big ec2 instance. The data I'm talking about will easily be around half a terabyte or more and I need efficient random access within it. At least initially. I assume that in the medium term this amount of data will grow quite a bit larger. I really do just want to use something like Tokyo Tyrant, but the data management overhead of sharding the datasets across enough high memory ec2 machines to get acceptable performance is prohibitive to going that route. I've kind of been driven to the dynamo / bigtable approach if only because it can store amounts of data in these sizes and can easily be ramped up by adding more servers.

That said, I'm still in the analysis stage of where our architecture needs to go next. It very well may be cost prohibitive to go with one of the technologies I mentioned.

Thank you for the benchmark tools and guidelines. They're immediately applicable to the analysis I'm performing this week.