My advice would be to give using a database for most of this and do everything you can in the web servers ram. Then spool the output to a disk / SSD array and/or a DB cluster.
Anyway, there is no way for people to read anywhere near this much data so you can just keep track of the data that's important to them in an approximate fashion. AKA if you give a data that's accurate +/- .2 seconds that's close enough.
Finally think about timestamps vs incrementing a counter so you can have each system independent of all the others and then aggregate the data when your done. (Merging 30 or so streams of timestamped data can be extremely fast.) And then add your post #.
Figure out how to partition your data/algorithms so you can split it across multiple machines, don't concentrate on maximizing the performance of one machine, but rather of the "cloud" of machines.
(/ (* 150000 ; users
3.5 ; times per min
(+ 5 ; cookie length
7.5 ; chars per post
)
)
(expt 10 6) ; bytes per MB
)
Evals to ~6.6. Double that (for max capacity estimate) = 13.2. Unit is MB/min, which is ~0.22 KB/sec. One hour is just short of 800 MB of plain text data. A gig of DB seems reasonable (I don't know what extra overhead is associated with, say, a CHAR(15) field).Like Rectic said, a RAM based database seems better.
PS sorry, I don't know how to format code here.
The 'other post' changes from time to time based on an algorithm that takes into account all the currently existing posts. This means I must determine the position of the 'other post' each time, and then relate it to the position of the member's new post. Once I know their relative positions I can return the correct positioning data to the member.
I would suggest you load test extensively and make load testing a part of your development process from the get-go. Initially, I would load test to evaluate approaches and estimate hardware needs. As you develop, I would use load test each version and ultimately simulate a real world scenario before letting it loose. Rather than speculating, you can actually know what approaches work best, and have some guarantee the live system will work as required.
The concept is a live interactive system with 100,000+ members online, logged in, and actively posting for about an hour at a pre-determined time of the day (or night). Each person may post 2-5 times a minute and this can add up to 1/2 million hits a minute, but I said I need a million hit per minute capacity because it is possible that 200,000 people might be participating in some of these events. This is a high-profile venture and "bogging down" would be an undesirable occurrence in the middle of an event.
Each post from a logged in member will contain a value which typically consists of no more than 5-10 characters. A cookie of 5 characters will accompany the posted data to uniquely identify the visitor. When the server receives this data it will append a new record to the database with the post value in one field, the login cookie in another, a date/time stamp in the third field, and an incremented value in the fourth field to identify the order in which each post is received. I think that's all the data that needs to be appended to the db in each new record.
(Note that this is my current concept of the database schema, if you folks know a better way to do it please tell me, thanks.)
After the new record is appended, a search is performed and the post field is sorted, then the relative position of the visitor's new database record is identified in the results (example 49,501 records from the top). This 'position' information is used by the script that dynamically generates the HTML, which is then passed off to one of the HTTP servers so it can return the page to the visitor's browser.
Since the returned data must be created dynamically based on the most recent database update, it appears that the database itself may be the limiting factor here. I can equip the database server with SSD drives to accelerate its performance but this still may not be fast enough. Plus there may be an issue with database size itself since it will grow extremely rapidly with 16,000 new records appended every second for an entire hour ...
I have not calculated the size of such a database after an hour of such appends, and I don't know whether existing SSD drives have the capacity needed to store the volume of data appended at this rate, or even if today's database systems can use multiple SSD drives if more capacity is needed. These are issues I may have to deal with if something else does not come up first that makes the whole concept look absolutely impossible.
And yes, I am absolutely going to need this capacity from the start. This is not a system I can "grow into", it must be capable of this performance from the very beginning. Assuming that such a system can be created its cost may not pose a problem since the concept will be very attractive to investors based on the huge revenues it can potentially generate. If there is no practical solution at this time I'll have to re-think the entire concept and try to come up with an alternative that works within today's server / database / page rendering limitations. I'm just hoping there's an existing solution so I don't have to change the concept because as-is it is unique and has unbelievably huge upside potential.