It's said before, but I want to expand a little bit. You're getting 10MB/sec because your volume is running as a virtual disk on a SAN system which runs on 100s on disks which can read 15+ GB/sec, but serving to 1000 servers at the same time.
If you had local storage, you'd have the same speed because your storage backbone would have the same throughput/consumer ratio.
You want faster drives or higher throughput/consumer ratio? You'd need faster RAID cards with faster drives and more PCIe lanes.
Guess what? This is enterprise hardware again :)
> For many uses, users don't care about true reliability.
Reliability does not always concern with noisy problems (data loss, crash, file loss, etc.). The biggest problem is silent corruption. A wrongly compiled, somewhat buggy binary. A slightly wrongly trained model. A wrong outcome from a computation or simulation.
Lost files are not a problem. Wrong outcomes looking right is.
That's because they're not bare hard drives directly connected to the server, they're a virtual hard drive over SAN.
Many modern amd offerings support ecc memory, but I can't comment on intel.
> Consumer grade HDDs are slow.
Meanwhile cloud provider hdd storage is slower than usb2. I have a 32 core machine deployed in azure right now, and the hdd that was deployed with it has a max read speed of 10MB/s. Getting a medium performing SSD seems to cost in the realm of $250/month just for the storage, before computer costs.
For many uses, users don't care about true reliability. I don't care if my CI agent has data loss, I'd _much_ much rather it ran on consumer grade hardware at speed and failed 2-3% of the time than what I've got now (as long as I have the choice. I want my build servers to run on fast unreliable hardware, for example)
Agreed re: pcie though!