News for hackers. Stuff that matters.
Re:Ask HN: How do I handle a million hits a minute? by Retric on Wednesday November 12, 2008 @09:43AM (#361839)
1) Skip SSD and keep the database in RAM. 2) Your not going to sort things that fast for every new post. 3) Think cheep HW that I can add as the system grows not a single beastly system that's fine for 150,000 users but breaks at 200,000 etc.

My advice would be to give using a database for most of this and do everything you can in the web servers ram. Then spool the output to a disk / SSD array and/or a DB cluster.

Anyway, there is no way for people to read anywhere near this much data so you can just keep track of the data that's important to them in an approximate fashion. AKA if you give a data that's accurate +/- .2 seconds that's close enough.

Finally think about timestamps vs incrementing a counter so you can have each system independent of all the others and then aggregate the data when your done. (Merging 30 or so streams of timestamped data can be extremely fast.) And then add your post #.

Re:Ask HN: How do I handle a million hits a minute? by ig1 on Wednesday November 12, 2008 @06:04PM (#362446)
I agree I've worked on a number of high volume systems (million+ client interactions/minute), and you don't want a conventional database. Either use a custom data-structure to keep it in memory (even if it's across multiple machines) or if you really want to use a database use one thats designed for that kind of usage (think tickerplant databases, kx, etc.)

Figure out how to partition your data/algorithms so you can split it across multiple machines, don't concentrate on maximizing the performance of one machine, but rather of the "cloud" of machines.

Re:Ask HN: How do I handle a million hits a minute? by frisco on Wednesday November 12, 2008 @12:12PM (#362011)
Also, what processing is actually going on with the data? Just updating the data in storage seem like less than half the problem. You're going to need an efficient system for parallelizing whatever processing jobs need to be run.