It seems to me that their business model, and their requirements, and their implementation solve one problem, and they do it well. The problem backblaze is solving is backups. Their front-end software sends a pile of data to the datacenter, where it is written en-masse. Should restore ever be needed, the data is then transfered back en-masse. Both of these operations have needs that are different from an S3 type service. Backup/restore can have latencies built in, can have slow seek times, etc. This allows them to take advantage of the lower end hardware.

For example each of these "pods" only has 4GB of RAM. If I was doing lots of random I/O on 67TB of drives, I would want a heck of a lot more ram for caching efficiencies.