News for hackers. Stuff that matters.
Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by sophacles on Thursday September 03, 2009 @03:20PM (#803268)
Ok I must be missing something. Everyone is knocking this backblaze thing because it doesnt do ZFS or because it is not a "super duper high end san replacement" or because the components are not rated for enterprise level work. It seems to me that a company whose product is personal and small business backups does not need any of those things. They have software to do mirroring. Why would they need any fast network filesystems if all data comes and goes via the internet? Will the hard drives see that much data I/O, or will they mostly just fill up and sit idling with the very occasional read for restore (i strongly suspect the later)[1].

Of course on the other hand, there are the poeple touting it as the end all be all, a solution to kill NetApp.

I guess what Im wondering is: how did so many people get the idea that this article about a specific solution to a specific problem was actually some sort of general purpose solution attacking all the big name people? What am I missing?

[1]A huge chunk of this article is about the hard drives and PSUs not being enterprise ready, but for an enterprise load, but I just don't see it. I bet a lot of these boxes run idle a large chunk of the time. I have 10 year old desktop hard drives running just fine in a file server, because it has a similar load: mostly idle most of the time.

Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by uibuibk on Thursday September 03, 2009 @05:31PM (#803612)
Exactly - this is a replacement for a tape library, not a database server.
Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by jrg on Friday September 04, 2009 @06:10AM (#804572)
Indeed. It would be nice to see some comparisons to the architectures (and, as everyone seems to care so much, prices) in the various 'VTLs' sold by the likes of Falconstor, HP, EMC, and StorageTek/Sun!
Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by rbanffy on Thursday September 03, 2009 @06:19PM (#803698)
"how did so many people get the idea that this article about a specific solution to a specific problem was actually some sort of general purpose solution attacking all the big name people?"

Backblaze did it, They did it when they decided to put this graph in the article:

http://blog.backblaze.com/wp-content/uploads/2009/08/cost-of...

It's an apples to oranges comparison.

Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by russss on Thursday September 03, 2009 @07:40PM (#803848)
Well no, it's not an apples to oranges comparison because no hardware vendors offer a solution which is comparable to the one Backblaze have devised.

That graph is simply comparing their solution to the closest commercial equivalents. It just so happens that these are all way off because all hardware manufacturers want to design their hardware to provide the highest throughput.

(That said, our experience with the Sun X4500s hasn't been great from an I/O point of view.)

Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by Periodic on Thursday September 03, 2009 @08:18PM (#803935)
It is an apples to oranges comparison because Blackblaze is only including the cost of components in that graph for their solution, but they're including all of the research, development, assembly, and support costs in the other vendors' solutions.

What are the labor costs for testing all their hardware (with 10 sata controllers no less)? What are the labor costs for assembling all those systems? What is the labor costs of developing the system architecture? What are the labor costs of creating their storage application which reduces their need for in-box redundancy? And if you want to talk about Amazon there is data center rental, power, cooling, and ongoing administration.

In short, they're comparing the costs of their components to the costs of a ready-to-go solution from other vendors. Sounds like apples and oranges to me.

Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by easp on Tuesday September 08, 2009 @01:05PM (#811388)
They did invite these criticisms by drawing comparisons to S3, EMC, etc. However, they already tried to factor out the cost of operations so that s3 was on equal footing with the non-service offerings. We can quibble about the specific numbers they used, but I think the bigger quibble is that by their own estimates, those costs are substantial, seemingly far outstripping their hardware costs by something like an order of magnitude. Even so, at the scale of a pentabyte or more, the savings are substantial enough to be worth addressing.

As for the missing costs that you cite, think this through. What do you really think the per-unit costs for assembly and testing are? Even if each unit required a couple of man-days, the costs would probably only add ~10% or so per unit. The other costs you cite for the initial hardware and software engineering are fixed costs, the same whether they are storing 1PB or 1,000. Also, if they did their job right, much of the per-unit testing cost should be mitigated by the overall systems design. The system management automation can do a test cycle on new nodes before promoting them to production use, and of course, failures should be dealt with automatically.

Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by russss on Friday September 04, 2009 @11:21AM (#804960)
Good point, but I suspect even given the amount of money they've spent on development, and given the production runs on these things are tiny, they'll probably still beat Sun on total price. Hell, even if they only made one of the things they could spend $800k on developing/manufacturing it and still beat Sun.

As far as I'm concerned, you'd have to write the custom software even if you used Sun's boxes. I wouldn't trust data to a single machine of anyone's design. Though that's probably just me.

The S3 comparison is totally invalid though, I agree.

Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by rbanffy on Thursday September 03, 2009 @09:18PM (#804040)
"That said, our experience with the Sun X4500s hasn't been great from an I/O point of view."

Of course it's not. It's an IBM PC down inside. I bet it can still boot MS-DOS and run GW-BASIC. ;-)

PCs are not a nice architecture for servers: there are starvation points all over the system, from registers (AMD64 solves some of it) to memory to I/O. Sun could have based the Thumper on a more server-ish (SPARC?) design, with plenty of memory and I/O bandwidth, but then it would not run Windows and the ability to run Windows is a defining advantage in the high-volume server market.

If I were to design such a box from scratch, I would couple the disks close to the network interfaces over a dedicated bus so the CPU could just say something like "hey, disk 3, drop blocks 10239 through 10300 on buffer 12 of your network controller while I go on with header parts and prepare to send it off". While I am at it I would skip the PSUs and go with DC power and a small battery, Google-style.

I fear I probably described something Thumper-ish and I will be ridiculed by someone with lots of server design experience, but that's the life of a hardware engineer who went the software very early. And I welcome such criticism ;-)

Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by nkurz on Friday September 04, 2009 @12:33AM (#804280)

  PCs are not a nice architecture for servers: there are
  starvation points all over the system, from registers (AMD64
  solves some of it) to memory to I/O.
Could you clarify the ways in which you think a PC architecture will hobble a data server? It looks like the X4500 runs dual Opterons, which relatively speaking have no shortage of registers. And it supports 16 GB of memory, which doesn't seem too shabby. And while I'm not sure how the SATA controllers are connected to the bus, I'd be a little surprised if this was the bottleneck. Which part of the 'PC architecture' do you think is the limiting factor?
Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by rbanffy on Friday September 04, 2009 @07:35AM (#804628)
- It's not the amount of memory, it's how much memory you can transfer without disrupting program execution due to bus contentions. It makes a huge difference whether your 16 GB of memory are in two, four or eight sockets (different sockets can (or at least could) be accessed simultaneously) or if it's attached to a single processor or pooled system-wide.

- AMD64 is a little better than vanilla 32-bit x86, but it still has few registers compared to POWER or SPARC architectures. This increases the risk of memory access, which is bad. I suppose there is a point when it's pointless to add registers and x86s do some convoluted stuff with shadow registers, so the picture is not really clear. Optimizing compilers should alleviate this too, but, like car builders say, there is no substitute for the cubic-inch.

- Still about processors, the least a multi-threaded CPU can do for you is to keep an execution context in-chip and prevent a context swap from memory. That saves a lot of memory bus time that cannot be used by other parts of the system. AMD64s (and their Intel counterparts, AFAIK) max out at 2 threads/core. POWER and SPARC max out at 4 and 8 tpc respectively (again, a number off the top of my head).

- On PCs (defined here as "a computer that can run Windows"), there is little distributed intelligence. I never saw a PC where CPU, disk controller and network interfaces could do the chat I described. Contrast it with the typical vintage mainframe design, where there are as many things going in parallel as designers can think of. As as example, there is an IBM disk-drive in the Computer Museum where you can see two sets of heads/arms on opposing sides of the disk, effectively being able to read/write different cylinders simultaneously. While I don't believe such machines are in current use, this serves to illustrate how far a server designer is willing to go in order to beat a throughput record.

Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by pmjordan on Friday September 04, 2009 @06:10AM (#804573)
It's an apples to oranges comparison.

Maybe for the low-level tech, but certainly not what it's used for. There are lots of online backup services out there. Some of them publicly make it known they use Amazon S3 for storage. I'm sure there are some other companies, who own their hardware, like Backblaze, but didn't optimise their cost structure using custom builds.

Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by gaius on Thursday September 03, 2009 @03:55PM (#803373)
If "doing backups" is your business, then "look how cheap our kit is!" probably isn't the best marketing strategy. A "small business" is someone's bread-and-butter, they don't care that their backup needs aren't sexy to storage geeks.
Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by sophacles on Thursday September 03, 2009 @04:05PM (#803397)
I dunno, it made me consider them. Instead of "we spend $2m on storage, and someday to hope to make enough money from people like you to not go out of business", its "look we aren't charging you much, and we are promising you 50 yrs of storage, this is how we can actually do that and stay in business".
Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by sho on Friday September 04, 2009 @12:59AM (#804302)
I agree. Do you remember a company called Bandwagon? http://ridethebandwagon.com/

They had a big splash a couple of years ago about how they were going to offer unlimited backup for all your mp3s in iTunes, all nicely and automatically synced, for about $10/month or something. They were going to do the storage on S3. They offered anyone who mentioned them on their blog a free years' service.

http://blog.ridethebandwagon.com/post/37269480/free-accounts...

I had about 400G of mp3s at the time I think. I thought, "OK!"

Their service lasted about 2 days as they underestimated, seemingly by about an order of magnitude, how much data people actually had.

http://delong.typepad.com/sdj/2007/02/adverse_selecti.html

Then they went back to the drawing board and came up with some lame service where you buy the storage space on S3 yourself and then they sell you the ability to access it or something. Lucky they didn't go out of business.

Anyway, knowing the details of the kind of system Backblaze is using, and the price per gig it breaks down to for them, is evidence they're not overselling themselves, and gives me confidence that they'll still actually be there in a few years, unlike Bandwagon. I'm much more likely to use them because of that. Add in all the discussion they've generated, and it's been a very clever marketing event for them. Kudos!

Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by fauigerzigerk on Friday September 04, 2009 @02:59AM (#804434)
The answer to your question is in the price comparisons that Backblaze provides. If you compare the price of something you imply that it fills the same or similar needs. Since there is no explicit caveat, and they don't explain why their solution is so much cheaper, competitors will come out and provide that explanation for them.

I'm pretty sure that the Backblaze solution is actually cheaper and that they are cutting some of the right corners for their particular application scenario. But it remains an open question how much cheaper it ends up being.

I think their price comparison is a bit misleading since it doesn't include replacement costs for the desktop grade hard disks they use and it doesn't include any of the costs incurred by the software/labor/maintainance required to make this a reliable backup solution.

Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by mmt on Friday September 04, 2009 @11:19AM (#804952)
Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by ciupicri on Friday September 04, 2009 @08:55PM (#806011)
Interesting papers, but based on what I've skimmed and remember I don't think that they compare desktop grade disks with server grade disks.
Re:Sun engineer responds to the Backblaze "Petabytes on a bu... by mmt on Tuesday September 08, 2009 @06:41PM (#812131)
Given the length of history, they do, in effect, by comparing disks with server-grade interfaces (e.g. SCSI) with desktop-grade interfaces (e.g. IDE).