But then, moving to flash is sort of a complicated version of the same solution...
And in any case, I said "complicated", not "cheaper", and I'll stand by that. Complexity has costs all by itself.
So I say it's cheaper and less complex.
Swapping hardware is a really expensive expensive change in the IT world, significantly more so than deploying new hardware on an existing infrastructure. That you somehow think otherwise is surprising to me.
If X is 100%, the approaches converge... but I think lots of things (popularity of songs, discretion over next song played, choice of N, etc.) could make buffers much smaller than total catalog size helpful in seek-reduction.
(The buffer could also be read-write SSD instead of RAM. Or perhaps just for the most popular items or skip-prone early ranges of files.)
Even with knowledge of the accesses required in 1s time, you're still bounded by the speed of the movements of the heads.
That said. I wonder if there's any trickery you can do with pre-generating the interleaved data for a group of users.
As in:
1) Read the next 5 (or any amount really) tracks for each of the users in a group.
2) Interleave the 16KB blocks in the same pattern you'd be streaming.
Step 1 is just a whole set of sequential reads. Step 2 is one big sequential write. Then your streaming operation is a single sequential read.
Of course, there's a good chance the overhead on doing the above outweighs the gain.
(I used to work for Last.fm and we spent a fair amount of time working on this issue without a great amount of success.)
I'm also wondering what moved them towards the SLC-based Intel drives as opposed to the MLC-based ones (that offer a much lower price-capacity), since in terms of read performance IOPS they're almost identical.
Also, if we ever decide to re-commission these machines later, the SLC based ones will be more useful for us.
X25-M: 1,200,000 Hours X25-E: 2,000,000 Hours
I'd be willing to replace them more regularly for the added capacity though.
At that price you can buy 3 X25-Ms and still have more space.
The real difference is in write cycles though: the X25-E can do ten times as many as the X25-M.
Much of the time, and oftentimes for runs of many minutes/megabytes at a time, you can predict exactly what data will be needed at what future time.
Perhaps another approach would be to add more buffering in front of spinning disks -- perhaps RAM, perhaps SSD -- and tune the disk IO scheduler to be extreme in the other direction: have the app request everything at least N minutes in advance, and let the scheduler take up to N minutes to deliver it. Then it can take advantage of any seek-minimizing physically-near blocks over a much larger window of time.
(For those occasional skip exceptions, data could be pulled from a smaller lower-latency system, or from a smaller set of 'whatever happens to be ready' fill-ins, etc.)
i.e. it reads 16KB of one track, sends it to one user, then 16KB of the next track, sends it to the next user, etc. Then after 1 second it goes through and does the whole thing again.
This is a massively seek-heavy operation.