News for hackers. Stuff that matters.
Re:Ask HN: How do I handle a million hits a minute? by owkaye on Wednesday November 12, 2008 @09:28AM (#361818)
Wow, I was writing a reply to davidw's first post when lots of new posts arrived in this thread. Thanks to all of you! I haven't read any of your new posts yet but I will as soon as I submit this one:

The concept is a live interactive system with 100,000+ members online, logged in, and actively posting for about an hour at a pre-determined time of the day (or night). Each person may post 2-5 times a minute and this can add up to 1/2 million hits a minute, but I said I need a million hit per minute capacity because it is possible that 200,000 people might be participating in some of these events. This is a high-profile venture and "bogging down" would be an undesirable occurrence in the middle of an event.

Each post from a logged in member will contain a value which typically consists of no more than 5-10 characters. A cookie of 5 characters will accompany the posted data to uniquely identify the visitor. When the server receives this data it will append a new record to the database with the post value in one field, the login cookie in another, a date/time stamp in the third field, and an incremented value in the fourth field to identify the order in which each post is received. I think that's all the data that needs to be appended to the db in each new record.

(Note that this is my current concept of the database schema, if you folks know a better way to do it please tell me, thanks.)

After the new record is appended, a search is performed and the post field is sorted, then the relative position of the visitor's new database record is identified in the results (example 49,501 records from the top). This 'position' information is used by the script that dynamically generates the HTML, which is then passed off to one of the HTTP servers so it can return the page to the visitor's browser.

Since the returned data must be created dynamically based on the most recent database update, it appears that the database itself may be the limiting factor here. I can equip the database server with SSD drives to accelerate its performance but this still may not be fast enough. Plus there may be an issue with database size itself since it will grow extremely rapidly with 16,000 new records appended every second for an entire hour ...

I have not calculated the size of such a database after an hour of such appends, and I don't know whether existing SSD drives have the capacity needed to store the volume of data appended at this rate, or even if today's database systems can use multiple SSD drives if more capacity is needed. These are issues I may have to deal with if something else does not come up first that makes the whole concept look absolutely impossible.

And yes, I am absolutely going to need this capacity from the start. This is not a system I can "grow into", it must be capable of this performance from the very beginning. Assuming that such a system can be created its cost may not pose a problem since the concept will be very attractive to investors based on the huge revenues it can potentially generate. If there is no practical solution at this time I'll have to re-think the entire concept and try to come up with an alternative that works within today's server / database / page rendering limitations. I'm just hoping there's an existing solution so I don't have to change the concept because as-is it is unique and has unbelievably huge upside potential.

Re:Ask HN: How do I handle a million hits a minute? by Retric on Wednesday November 12, 2008 @09:43AM (#361839)
1) Skip SSD and keep the database in RAM. 2) Your not going to sort things that fast for every new post. 3) Think cheep HW that I can add as the system grows not a single beastly system that's fine for 150,000 users but breaks at 200,000 etc.

My advice would be to give using a database for most of this and do everything you can in the web servers ram. Then spool the output to a disk / SSD array and/or a DB cluster.

Anyway, there is no way for people to read anywhere near this much data so you can just keep track of the data that's important to them in an approximate fashion. AKA if you give a data that's accurate +/- .2 seconds that's close enough.

Finally think about timestamps vs incrementing a counter so you can have each system independent of all the others and then aggregate the data when your done. (Merging 30 or so streams of timestamped data can be extremely fast.) And then add your post #.

Re:Ask HN: How do I handle a million hits a minute? by ig1 on Wednesday November 12, 2008 @06:04PM (#362446)
I agree I've worked on a number of high volume systems (million+ client interactions/minute), and you don't want a conventional database. Either use a custom data-structure to keep it in memory (even if it's across multiple machines) or if you really want to use a database use one thats designed for that kind of usage (think tickerplant databases, kx, etc.)

Figure out how to partition your data/algorithms so you can split it across multiple machines, don't concentrate on maximizing the performance of one machine, but rather of the "cloud" of machines.

Re:Ask HN: How do I handle a million hits a minute? by frisco on Wednesday November 12, 2008 @12:12PM (#362011)
Also, what processing is actually going on with the data? Just updating the data in storage seem like less than half the problem. You're going to need an efficient system for parallelizing whatever processing jobs need to be run.
Re:Ask HN: How do I handle a million hits a minute? by spinonethird on Wednesday November 12, 2008 @11:10AM (#361924)
From your description you don't seem to really use the db, so can't you roll your own specialized data structure? If each record is 20 byte, 1/2 million per minute for one hour is about 1/2 gig, so you can keep everything in memory.
Re:Ask HN: How do I handle a million hits a minute? by whacked_new on Wednesday November 12, 2008 @10:59AM (#361910)
No scaling guru here, but did this quick calculation for fun (gotta love C-x-e):

(/ (* 150000 ; users

      3.5 ; times per min

      (+ 5   ; cookie length

         7.5 ; chars per post

         )

      )

      (expt 10 6) ; bytes per MB

      )
Evals to ~6.6. Double that (for max capacity estimate) = 13.2. Unit is MB/min, which is ~0.22 KB/sec. One hour is just short of 800 MB of plain text data. A gig of DB seems reasonable (I don't know what extra overhead is associated with, say, a CHAR(15) field).

Like Rectic said, a RAM based database seems better.

PS sorry, I don't know how to format code here.

Re:Ask HN: How do I handle a million hits a minute? by lacker on Wednesday November 12, 2008 @12:15PM (#362015)
This doesn't really make sense. If you have 100,000+ members online, each posting at least every 30 seconds, you are going to have way more user-provided data than you can possibly show to everyone. So why do you need to know the exact relative position of the visitor's new database record?
Re:Ask HN: How do I handle a million hits a minute? by owkaye on Wednesday November 12, 2008 @12:41PM (#362042)
I'm not showing them all that user-provided data, all I'm showing them is the position of their last post relative to the position of 'another post' in the database.

The 'other post' changes from time to time based on an algorithm that takes into account all the currently existing posts. This means I must determine the position of the 'other post' each time, and then relate it to the position of the member's new post. Once I know their relative positions I can return the correct positioning data to the member.

Re:Ask HN: How do I handle a million hits a minute? by cstejerean on Wednesday November 12, 2008 @02:51PM (#362222)
The details of what exactly is important for the purpose of determining the "other post" can significantly impact how you need to store the data and the performance implications of doing so. Does it depend on the time the request is made or just based on the data in the database? Can you determine approximately when that other post is likely to change or do you have to try to compute it on every request? Is the "other post" the same for all the users? For a subset of users? Is it unique for every user?
Re:Ask HN: How do I handle a million hits a minute? by lacker on Wednesday November 12, 2008 @02:04PM (#362155)
This logic sounds like the important part of your application, but I don't get it yet. Why does the user care about the position of their post relative to some other post? Could it be approximate instead? Also, does it have to be accurate up to the second, or would a slower update cycle work? I feel like maybe you could make some very small changes to the algorithm, and make it much more tractable on the backend.
Re:Ask HN: How do I handle a million hits a minute? by owkaye on Thursday November 13, 2008 @02:58AM (#362856)
I've created a (hopefully) more organized thread with some details this one does not have. Please go to this thread to continue this discussion, thanks:

http://news.ycombinator.com/item?id=362810

Re:Ask HN: How do I handle a million hits a minute? by ericb on Wednesday November 12, 2008 @01:59PM (#362144)
This is not a system I can "grow into", it must be capable of this performance from the very beginning.

I would suggest you load test extensively and make load testing a part of your development process from the get-go. Initially, I would load test to evaluate approaches and estimate hardware needs. As you develop, I would use load test each version and ultimately simulate a real world scenario before letting it loose. Rather than speculating, you can actually know what approaches work best, and have some guarantee the live system will work as required.

Re:Ask HN: How do I handle a million hits a minute? by mseebach on Wednesday November 12, 2008 @10:27AM (#361879)
What exactly is the output?