Re:Coding for SSDs
by danielvinson on Friday April 24, 2015 @11:59AM
(#9434199)
BootRacer tests are really inaccurate. The problem is that boot times are dependent on many things outside of the SSD, namely Chipset, drivers, and BIOS settings. Testing methodology for the manufacturer would be to disable all features in the BIOS for this test to try to reduce variables, which leads to unclear lower numbers.
The first rule of benchmarks is to only change one thing at a time. Any test that changed more than just the drive would not be accurate.
Wouldn't it make sense to keep the same hardware for all of one series of tests except the SSD?
Do you have any links that would show that an SSD was tuned solely for benchmark purposes? I know people have found such things in GPU market but I've never heard about such claims in SSD market.
"Back in the days many SSD vendors were only focusing on high peak performance, which unfortunately came at the cost of sustained performance. In other words, the drives would push high IOPS in certain synthetic scenarios to provide nice marketing numbers, but as soon as you pushed the drive for more than a few minutes you could easily run into hiccups caused by poor performance consistency. Once we started exploring IO consistency, nearly all SSD manufacturers made a move to improve consistency."
http://www.anandtech.com/show/9144/crucial-bx100-120gb-250gb...
Sorry I can't provide links. But I work for a large tech company, currently qualifying SSDs for a new project. We ran into serious bugs with drives from a particular well-known SSD manufacturer (certain workloads cause drives to become unresponsive until power is cycled) and were able to demand detailed root cause analysis. In at least two cases they uncovered new bugs in code that had been inserted purely to improve the numbers for 4k random writes on certain benchmarks. These are consumer drives, not enterprise, and are current production.
I've had that much acknowledged from SSD vendors, each one blames the others for the benchmark wars but they all know that benchmarks are out there and customers will chose the SSD based on the published benchmarks so they tune the firmware to make sure they don't look bad in the benchmarks.
I was even told that an SSD was made to work consistently over the life span but drop some of these consistency work (i.e. slow downs at start of life) and only kick them in after so many power-on-hours so that it will not look bad in benchmarks.
It's a fact of life that customers rely on benchmarks and that vendors cannot educate all customers so the masses out there need to see a good benchmark so that's what gets optimized for.
There's a lot of complexity going on behind the scenes of modern SSDs. Simplistic benchmarks as posted on most review sites often don't address real-world performance. The ones I pay most attention to are things like BootRacer; it's quite remarkable how often a drive will be slower at booting Windows than a competitor, even when it beats them in simple metrics like sequential / random reads / writes with different queue depths.
I can see the potential for firmware being optimized for modalities that crop up often in benchmarks but less often elsewhere: the aforementioned sequential / random reads and writes with different block sizes and queue depths. Doing well on benchmarks probably drives a bunch of sales. But detecting and switching modality may not be latency-free, and shortcuts taken to improve absolute performance in those modalities may harm global performance in a real-world IO mix.