News for hackers. Stuff that matters.
Re:Ask HN: Which persistent in-memory database to use? by paraschopra on Tuesday May 26, 2009 @11:44PM (#627951)
Hey, Thanks for the pointers.

I estimate Reads to be 10x more frequent than writes and Reads would mostly be a key lookup, with occassional aggregation.

Yes, I was also wondering if anybody has demonstrated sclability with MySQl for this level, as I won't have to fiddle around with code too much.

Re:Ask HN: Which persistent in-memory database to use? by bsaunder on Wednesday May 27, 2009 @08:17AM (#628357)
That sounds rather normal (reads 10x writes and 100s/1000s per second). As far as I understand MySQL can definitely scale to that. It does take considerable effort to configure and tune it (but any database solution will perform horribly out-of-the-box). Twitter uses MySQL. I'd doubt that you will hit their level of performance (though perhaps you are hoping to ;).

I think you should read up on using Memcached with MySQL.

Here's a good starting point: http://highscalability.com/bunch-great-strategies-using-memc...

At this point in the game (it sounds early in your development process), you aren't painting yourself in a corner (the only thing you should be worried about here IMHO) if you go with these two, but you will probably have to refine things later on to get your scalability (again, that's true with any solution you pick).

Also realize that Twitter probably has a larger hardware budget than you do and real scalability costs real money in both hardware and effort. It's been widely reported that many of their engineers have been spending considerable effort keeping things stable and improving scalability. Another performance related mantra is that "to go faster, do less". This is definitely true here... there may be things you think you need to store, but don't. Removing that effort could have a profound impact on performance.

I suppose its worth mentioning that you may want to check out AWS and their storage solutions (and push the performance problems off to them). This decision also has other business factors (like can you deal with potential outages, potential security problems, cost, etc...).