Re:FlockDB: Twitter's distributed, fault-tolerant graph data...
by lsb on Monday April 12, 2010 @10:28AM
(#1259180)
You can go a long way with a SQL database too. I'm writing up an article about getting the two billion words of Wikipedia into an inverted index in a SQLite database entirely in memory, and that's another other order of magnitude bigger.
Re:FlockDB: Twitter's distributed, fault-tolerant graph data...
by amix on Monday April 12, 2010 @09:07PM
(#1260901)
If you only know how to use a hammer then everything else looks like a nail. In other words, re-implementing an inverted index in SQL is waste of time when you can use tools like Sphinx and Lucene - which are highly optimized to do inverted indexes and that can easily handle 2 billion words. The same can be said about FlockDB - it's possible to emulate a graph database, but is the effort really worth it when there are such tools like Redis and neo4j which seem to be optimized for graph like structures.
I think Redis would perform a lot better than SQL for graph like structures - since sets are a native datatype in Redis. And you can go A LONG way with just one Redis database (currently we are storing over 20 million keys in our Redis database and I know some that are storing 100 million keys on _one_ server). And with the new Redis VM coming up, I would guess that scalability of Redis is going to be even better.
Other than this, neo4j seems very interesting and would probably also have been a better choice than using a relational database.