Backblaze did it, They did it when they decided to put this graph in the article:
http://blog.backblaze.com/wp-content/uploads/2009/08/cost-of...
It's an apples to oranges comparison.
That graph is simply comparing their solution to the closest commercial equivalents. It just so happens that these are all way off because all hardware manufacturers want to design their hardware to provide the highest throughput.
(That said, our experience with the Sun X4500s hasn't been great from an I/O point of view.)
What are the labor costs for testing all their hardware (with 10 sata controllers no less)? What are the labor costs for assembling all those systems? What is the labor costs of developing the system architecture? What are the labor costs of creating their storage application which reduces their need for in-box redundancy? And if you want to talk about Amazon there is data center rental, power, cooling, and ongoing administration.
In short, they're comparing the costs of their components to the costs of a ready-to-go solution from other vendors. Sounds like apples and oranges to me.
As for the missing costs that you cite, think this through. What do you really think the per-unit costs for assembly and testing are? Even if each unit required a couple of man-days, the costs would probably only add ~10% or so per unit. The other costs you cite for the initial hardware and software engineering are fixed costs, the same whether they are storing 1PB or 1,000. Also, if they did their job right, much of the per-unit testing cost should be mitigated by the overall systems design. The system management automation can do a test cycle on new nodes before promoting them to production use, and of course, failures should be dealt with automatically.
As far as I'm concerned, you'd have to write the custom software even if you used Sun's boxes. I wouldn't trust data to a single machine of anyone's design. Though that's probably just me.
The S3 comparison is totally invalid though, I agree.
Of course it's not. It's an IBM PC down inside. I bet it can still boot MS-DOS and run GW-BASIC. ;-)
PCs are not a nice architecture for servers: there are starvation points all over the system, from registers (AMD64 solves some of it) to memory to I/O. Sun could have based the Thumper on a more server-ish (SPARC?) design, with plenty of memory and I/O bandwidth, but then it would not run Windows and the ability to run Windows is a defining advantage in the high-volume server market.
If I were to design such a box from scratch, I would couple the disks close to the network interfaces over a dedicated bus so the CPU could just say something like "hey, disk 3, drop blocks 10239 through 10300 on buffer 12 of your network controller while I go on with header parts and prepare to send it off". While I am at it I would skip the PSUs and go with DC power and a small battery, Google-style.
I fear I probably described something Thumper-ish and I will be ridiculed by someone with lots of server design experience, but that's the life of a hardware engineer who went the software very early. And I welcome such criticism ;-)
PCs are not a nice architecture for servers: there are
starvation points all over the system, from registers (AMD64
solves some of it) to memory to I/O.
Could you clarify the ways in which you think a PC architecture will hobble a data server? It looks like the X4500 runs dual Opterons, which relatively speaking have no shortage of registers. And it supports 16 GB of memory, which doesn't seem too shabby. And while I'm not sure how the SATA controllers are connected to the bus, I'd be a little surprised if this was the bottleneck. Which part of the 'PC architecture' do you think is the limiting factor?- AMD64 is a little better than vanilla 32-bit x86, but it still has few registers compared to POWER or SPARC architectures. This increases the risk of memory access, which is bad. I suppose there is a point when it's pointless to add registers and x86s do some convoluted stuff with shadow registers, so the picture is not really clear. Optimizing compilers should alleviate this too, but, like car builders say, there is no substitute for the cubic-inch.
- Still about processors, the least a multi-threaded CPU can do for you is to keep an execution context in-chip and prevent a context swap from memory. That saves a lot of memory bus time that cannot be used by other parts of the system. AMD64s (and their Intel counterparts, AFAIK) max out at 2 threads/core. POWER and SPARC max out at 4 and 8 tpc respectively (again, a number off the top of my head).
- On PCs (defined here as "a computer that can run Windows"), there is little distributed intelligence. I never saw a PC where CPU, disk controller and network interfaces could do the chat I described. Contrast it with the typical vintage mainframe design, where there are as many things going in parallel as designers can think of. As as example, there is an IBM disk-drive in the Computer Museum where you can see two sets of heads/arms on opposing sides of the disk, effectively being able to read/write different cylinders simultaneously. While I don't believe such machines are in current use, this serves to illustrate how far a server designer is willing to go in order to beat a throughput record.
Maybe for the low-level tech, but certainly not what it's used for. There are lots of online backup services out there. Some of them publicly make it known they use Amazon S3 for storage. I'm sure there are some other companies, who own their hardware, like Backblaze, but didn't optimise their cost structure using custom builds.
They had a big splash a couple of years ago about how they were going to offer unlimited backup for all your mp3s in iTunes, all nicely and automatically synced, for about $10/month or something. They were going to do the storage on S3. They offered anyone who mentioned them on their blog a free years' service.
http://blog.ridethebandwagon.com/post/37269480/free-accounts...
I had about 400G of mp3s at the time I think. I thought, "OK!"
Their service lasted about 2 days as they underestimated, seemingly by about an order of magnitude, how much data people actually had.
http://delong.typepad.com/sdj/2007/02/adverse_selecti.html
Then they went back to the drawing board and came up with some lame service where you buy the storage space on S3 yourself and then they sell you the ability to access it or something. Lucky they didn't go out of business.
Anyway, knowing the details of the kind of system Backblaze is using, and the price per gig it breaks down to for them, is evidence they're not overselling themselves, and gives me confidence that they'll still actually be there in a few years, unlike Bandwagon. I'm much more likely to use them because of that. Add in all the discussion they've generated, and it's been a very clever marketing event for them. Kudos!
I'm pretty sure that the Backblaze solution is actually cheaper and that they are cutting some of the right corners for their particular application scenario. But it remains an open question how much cheaper it ends up being.
I think their price comparison is a bit misleading since it doesn't include replacement costs for the desktop grade hard disks they use and it doesn't include any of the costs incurred by the software/labor/maintainance required to make this a reliable backup solution.
I believe the data dispel this marketing myth.
http://usenix.org/events/fast07/tech/schroeder/schroeder_htm...
http://usenix.org/events/fast07/tech/full_papers/pinheiro/pi...
Of course on the other hand, there are the poeple touting it as the end all be all, a solution to kill NetApp.
I guess what Im wondering is: how did so many people get the idea that this article about a specific solution to a specific problem was actually some sort of general purpose solution attacking all the big name people? What am I missing?
[1]A huge chunk of this article is about the hard drives and PSUs not being enterprise ready, but for an enterprise load, but I just don't see it. I bet a lot of these boxes run idle a large chunk of the time. I have 10 year old desktop hard drives running just fine in a file server, because it has a similar load: mostly idle most of the time.