This sounds reasonable. AFAIK the default readahead on Linux is 128KB, so you'll probably need to test and tune quite a bit to get it right. I had a paragraph roughly along those lines in an early draft, but decided it was too much detail. "Concurrent reads" is probably more accurate than plain "seek time".
Re:The Full Stack, Part I
by tedunangst on Thursday December 02, 2010 @05:35PM
(#1963955)
The kernel readahead is actually irrelevant. Your app issues 1MB reads one at a time and the kernel will fetch that all at once.
That's all fine and dandy till the users decide to pause, seek to a different spot in the stream or switch streams altogether rapidly.
When a user hits "next" to go to the next movie, it should start playing in less than a second or two. When a user seeks to a new chapter in the movie, the same thing.
Your math works if the users passively sit and consume a stream without much interaction.
That's the "put all 10TB on each streaming server" solution that he immediately rejected because of expense. You can see that the more streaming servers you connect to the storage system, the more important seek time becomes. He did then say "whatever architecture", which is not quite right since the seek bottleneck may not appear until the streaming:storage ratio gets higher than 1.
For a realistic exercise, throw a cost factor into your equation and optimize for cost and latency (aka user experience).
Re:The Full Stack, Part I
by tedunangst on Thursday December 02, 2010 @05:31PM
(#1963941)
In the big picture, are the drives that expensive? Your monthly bandwidth bill is going to come close.
Measuring latency (vs seek time) is definitely a better metric. But is a 5 second (the horror!) delay to watch a 40 minute video that bad for user experience?
A better example may have been streaming music, where we already know last.fm got a big win from using SSDs.
The drive themselves might be an extra 10 or 20% of the cost of the machine. They come with other factors as well. 5x the disks means more power draw and cooling. It probably also means you need a 2U chassis instead of pizza-box 1U, which means your datacenter rent doubles. (It may be possible to fit 10TB into a 1U. I've never tried it, but I suspect you'll have space / power / heat problems.)
2000 streams, 5 drives, that's 400 per disk. Let's say we have the world's worst disks, that can only do 10 seeks per second. 400 / 10 means we have to buffer 40 seconds of data per stream (per seek) and we have to read it in 0.1 seconds before moving to the next stream. 300kbps * 40s / 8 = 1500K of data. 1.5M / 60MB/s disk transfer takes 0.025 seconds, well under 0.1.
I guess that's alluded to by "non standard prefetching", but I don't think it's that advanced. Especially since in a streaming video application, the client software is already going to be doing buffering for you. The bottleneck is bandwidth.
Check my math please? :)