How to: Compete with Amazon S3 without Buying Hardware

Re:How to: Compete with Amazon S3 without Buying Hardware by adamsmith on Friday April 27, 2012 @04:29PM (#3900900)

> This article is basically advocating RAID 5 across many storage providers.

That is correct. And to be more precise, I'm advocating RAID 5 across storage providers as a service, so people who just want to store data don't have to manage anything.

[ Parent | on HN ]

Re:How to: Compete with Amazon S3 without Buying Hardware by Terretta on Friday April 27, 2012 @04:41PM (#3900947)

>> This article is basically advocating RAID 5 across many storage providers.

> That is correct.

The diagram in the article and text description is RAID 1+0 aka RAID 10.

[ Parent | on HN ]

Re:How to: Compete with Amazon S3 without Buying Hardware by Dylan16807 on Friday April 27, 2012 @08:53PM (#3901665)

No, it's RAID 5. Well, specifically, it's RAID 6 but customizable. You can lose any k drives.

[ Parent | on HN ]

Re:How to: Compete with Amazon S3 without Buying Hardware by notmyname on Friday April 27, 2012 @06:32PM (#3901319)

If this were built on top of (or in conjunction with) Openstack swift (http://swift.openstack.org), it could be done with a common storage backend. There are already several public cloud storage providers using swift (Rackspace, HP, Softlayer, Internap, KT, and others), and a growing number of private deployments as well (Wikipedia probably being the most recognizable name).

If you'd like to talk more about this, send me an email (address in profile).

[ Parent | on HN ]

Re:How to: Compete with Amazon S3 without Buying Hardware by wmf on Friday April 27, 2012 @06:36PM (#3901337)

Aren't most Swift providers already using 2x-3x duplication? Putting parity on top of that would be like RAID15 (the reason you've never heard of RAID15 is that it would be crazy expensive).

[ Parent | on HN ]

Re:How to: Compete with Amazon S3 without Buying Hardware by notmyname on Friday April 27, 2012 @06:48PM (#3901364)

Yes, swift provides multiple redundant copies of the data. It depends on what you are protecting against: failure of a piece of your data or failure of a hosting provider.

If you simply rely on "dumb disks" spread across multiple providers to provide availability, then you may be interested in looking at the nova-volumes part of openstack (it provides block storage attached to nova VMs). As part of openstack, it's an open system that is seeing rapid adoption.

However, one of the most-requested features for swift is to provide support for logical clusters that span a wide geographic area. This could potentially allow multiple providers to collaborate in providing a multi-provider storage system. However, I'd guess that the technical problems are much simpler than the business problems in setting up multi-provider clusters.

[ Parent | on HN ]

Re:How to: Compete with Amazon S3 without Buying Hardware by thereallurch on Friday April 27, 2012 @04:35PM (#3900922)

Wouldn't the next issue become bandwidth? Sure, you have cheap storage, but you still need to manipulate the data somehow. I know S3 only charges for outbound data. What about the other companies? Is bandwidth free and plenty?

[ Parent | on HN ]

Re:How to: Compete with Amazon S3 without Buying Hardware by adamsmith on Friday April 27, 2012 @04:41PM (#3900950)

This is a really interesting question.

At scale, this could be managed, I think, through a combination of (a) shipping hard drives around, (b) caching, and (c) peering between storage providers. For example, shipping hard drives around would be expedient if you wanted to switch out a raw storage provider. The optimal strategy also depends on the access patterns and latency requirements.

It seems solvable, but not trivial.

[ Parent | on HN ]