News for hackers. Stuff that matters.
Re:Ask HN: Cassandra, HBase, Hypertable, Voldemort. Lowest r... by ghotli on Tuesday June 29, 2010 @09:32AM (#1471305)
Billions of records. I haven't yet determined which way to slice the dataset to be most efficient. A fair amount of it is denormalized and my assumption is that the initial dataset will be on the order of half a terabyte or more.

Right now the system I've conceived of has lookups by exact key, but the subsequent queries would be for data that is "near" to the previous query. I may mitigate that by increasing the size of data within the initial query to ensure that the "next query" is already loaded into memory.

The keys will always be consulted at random. That's kind of the problem.

Re:Ask HN: Cassandra, HBase, Hypertable, Voldemort. Lowest r... by gojomo on Tuesday June 29, 2010 @12:22PM (#1471875)
Only half a terabyte? All these distributed options are overkill.

Fit one machine with 5 160GB SSDs. Use anything. Classic BerkeleyDB or BerkeleyDB-JE should do.

Re:Ask HN: Cassandra, HBase, Hypertable, Voldemort. Lowest r... by ghotli on Tuesday June 29, 2010 @05:19PM (#1472890)
To begin with, yes. It also depends on how heavily I denormalize the data for recall. It could be many times that down the road. I'm not looking to implement something heavily distributed right now, but I know that day is coming with my datasets and I want to ensure I know what needs to be built when the time comes.