What database would you use to store (min) 10 billion objects?

Re:What database would you use to store (min) 10 billion obj... by scumola on Thursday May 07, 2009 @09:21PM (#598997)

That's approximately 200TB of data. I suggest sharding the data across multiple machines because a single SAN doing BDB lookups of a single file of that size is still going to be a tad slow. Most large databases nowadays are divided by sharding, but since it's key/value data, a BDB is probably one of the top solutions. If you modify a lot of the same data frequently, memcache can speed things up (also a key/value in-memory solution), but the size of the problem in total bytes may be more of a problem than the actual looking up and indexing of the data.

[ Parent | on HN ]

Re:What database would you use to store (min) 10 billion obj... by jaydub on Thursday May 07, 2009 @11:47PM (#599138)

but the size of the problem in total bytes may be more of a problem than the actual looking up and indexing of the data.

One cool property of such a large data-set (~200TB) is that you are pretty sure to see a lot of the same data repeated.

It'd be neat to try reduce the overall data footprint by assigning some sort of signature to repeats. I.e. if you are storing the sequences (let's imagine for a sec that these are non-trivial sizes) ABC, ABCD, ABCDE and ABCDEF , replace ABC with say #1 etc. and perhaps save a whole lot of space.

[ Parent | on HN ]

Re:What database would you use to store (min) 10 billion obj... by planck on Friday May 08, 2009 @03:46AM (#599332)

This is also known as the ZIP compression algorithm.

[ Parent | on HN ]

Re:What database would you use to store (min) 10 billion obj... by webignition on Friday May 08, 2009 @05:03AM (#599389)

But then you might end up not being able to see the forest of data for all the Huffman trees.

[ Parent | on HN ]

Re:What database would you use to store (min) 10 billion obj... by gaius on Friday May 08, 2009 @09:15AM (#599576)

It's called "deduplication" in databaseland, it's common strategy for tape backups.

[ Parent | on HN ]

Re:What database would you use to store (min) 10 billion obj... by gaius on Friday May 08, 2009 @06:27AM (#599448)

Most large databases nowadays are divided by sharding

Err, this isn't even close to true. MySQL isn't the only database in the world, you know.

[ Parent | on HN ]

Re:What database would you use to store (min) 10 billion obj... by sgk284 on Friday May 08, 2009 @08:20AM (#599531)

Sharding has nothing to do with MySQL, and everything to do with scaling.

[ Parent | on HN ]

Re:What database would you use to store (min) 10 billion obj... by didroe on Friday May 08, 2009 @08:35AM (#599539)

I think the point was that the big DBs have better ways to deal with scaling than sharding.

[ Parent | on HN ]

Re:What database would you use to store (min) 10 billion obj... by imajes on Thursday May 07, 2009 @09:55PM (#599023)

Yeah. but it's text, so i'm thinking about S3, interesting storage optimizations (alot of it would be tag data, for example,so there are plenty of fkeys).

i'd shard it based on some kind of distributed scheme... though not sure what that might look like. Good tips though. :)

[ Parent | on HN ]

Re:What database would you use to store (min) 10 billion obj... by scumola on Thursday May 07, 2009 @10:45PM (#599073)

Here's a good comparison of key/value database alternatives: http://www.metabrew.com/article/anti-rdbms-a-list-of-distrib...

[ Parent | on HN ]