Edit: by the way, I do have compression, thin provisioning, replication and it also auto provisions storage volumes when new VMs are created. As I said, I didn't post my two month old blog here and there is plenty of detailed information for those that are genuinely interested on its way in the next few weeks.
So you don't attempt to prejudice only in your blog posts, it appears. This is one of the most sadly defensive responses by a blog author I've yet seen on here.
Where, exactly, is the strawman? Please point it out rather than desperately trying to immunize against my completely valid comment in the most frantic of manners.
You posit an idealized notion about building it yourself based upon accomplishing, to this point, apparently very little. It's one thing to cast a theory and pursue it (e.g. "my attempt at building competitive storage on the cheap"), but you're presenting completely unsupported dogma around it, and then bolstering your own decisions by conclusions you actually don't even remotely have. It's the guy who decided to start working at a standing desk and on day one has a laundry list of comments about why anyone who doesn't is wrong and lesser.
This is a fairly typical, of course, and you see it by people who build their own anything ("don't give all your money to big TP -- six quick tips for making your own!"), and we generally only see it in the before stage, as the after stage is more often than not a littered debris field of failures.
I find it rather incredible that you attempt to incite HN guidelines, as if raw gullibility and boosterism was the direction of the critique. You are making broad claims in the linked article that you have absolutely no basis for making, so criticism is well deserved, and a service for anyone who might buy into this notion, making a fool of themselves in their own organization.
The strawman is that he builds a solution for his exact needs and you attack it for not having features (multipathing, replication, automatic tiering, thin provisioning, disk-deduplication, streaming hardware compression) that neither he nor most people need.
All that stuff becomes interesting when and if you need more than two bricks. In a time where you can fit >250T of spindles or ~42T of SSDs into a single server the audience for such shrinks rapidly.
I "attacked" (aka disagreed with) any broadly-targeted, generalist claims of the linked article (which is in stark contrast with, for example, the Backblaze posts where they build very purpose-suited storage and never try to over-extend their claims), which was quite clearly that people who buy so-called enterprise storage are, to paraphrase the gist of the article, suckers. I noted replication and multipathing because they are the absolute minimum cost of entry for critical storage, and even the linked article references it as a requirement.
The rest of the features were clearly "value adds", given that most enterprise storage features a lot of its value in software. Yes, everyone benefits from automatic tiering. Everyone benefits from thin provisioning, disk-deduplication, and compression. There are a vanishingly small number of users who won't see significant benefits from all of those features.
"All that stuff becomes interesting when and if you need more than two bricks."
? Multipathing is absolutely critical for a single storage unit. Replication is absolutely critical if you have any care at all about uptime, because a single storage unit, even with multipathing and redundant power supplies and "controllers", as appropriate, isn't enough. The rest of them have nothing at all do do with the number of storage units -- thin provisioning gives you fantastic storage control. Compression is obvious. Disk-deduplication...again, the name of the game is minimizing the amount of data you're actually dealing with, because even if have conceptually unlimited storage in your unit(s), that's data you have to move around and replication and backup and...
Most enterprise arrays these days are really software dressed up behind the hardware. As a customer, you get to pay for the hardware multiple times. This is really obvious in the backup space if you compare a software-only solution like CommVault to Avatar/DataDomain.
If you make the conscious decison to invest in engineering talent and have a lot of need for midrange storage, you can yield a positive return on that investment. If you're growing fast and go EMC, you'll invest significant capital in administering those solutions and negotiating anyway.
But if you don't have the resources to staff it, or the systems for provisioning/service management to replace the consoles you get from a vendor product, you don't belong in the business.
But this is exactly the sort of storage the linked article is talking about. So your point is that I attacked a "strawman" because I didn't accept the linked article for being something entirely different from what it is?
but the large majority of companies aren't
The majority of mid to large sized companies run entirely on SANs. I'm not sure where you get your information from, but how shoestring upstarts operate has nothing to do with much of the "real world".
This article is about a home-brew SAN, with home-brew SAN qualities and deficiencies.
(the essence was what you quoted: I claimed most people don't need these advanced SAN-features anymore because they don't need SANs in the traditional sense anymore)
Anyway, I'll just concede that OP aiming for feature-parity with commercial SAN's is indeed nonsensical.
The majority of mid to large sized companies run entirely on SANs
For large companies who bought into EMC/NetApp that may be true.
For small and mid-sized companies that's definitely not my experience.
Just because you've never seen one or had to deal with one directly does not mean you're not dependent on one.
I think the discussion was about running your own SAN, i.e. creating your own EBS.
As it happens EBS itself is not backed by a SAN either. According to the rumors it's implemented using either DRBD or GNBD.
However they provide a service. As the OP says its a bit broad to claim that they are a pointless cost. Having dealt with both, I can tell you that its very rare that own brand storage is cheaper in the long term. Unless: o you run at significant scale o have enough data to warrant a team of skill storage admins
bear in mind that decent storage admins cost the same as a medium sized array, per year, per person. Opex can quickly over take capex.
This is a baseless, prejudiced claim, contrasting buying with some idealized scenario where your in-house crew can only possibly build better solutions cheaper. That there are no other outcomes.
But they might also cost far more in manpower costs than any premium. They might give you an unreliable solution that costs you your entire business. Such an effort might distract from the core competencies of the organization. Operating that storage might end up dwarfing the up-front cost (it's easy to hire admins knowledgeable of EMC. Quite a different matter when it's your own home-brew solution).
I understand the draw, but the "all upside" claim undermines the entire piece. There are enormous downsides, not least the reliability of your data. Companies like EMC and Nimble -- despite "great" editorial quotes added by some random person on Wikipedia -- base their entire existence on reliably serving your data, and things like multipathing and replication are the absolute minimum cost of entry in the market. Now add automatic tiering, thin provisioning, disk-deduplication and streaming hardware compression, etc, and the value starts to become evident.
EDIT: The moderation through this is an abomination. HN shouldn't be overly critical, but nor should it pander patronizingly to some tripe because the author happened by.