News for hackers. Stuff that matters.
Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by skuhn on Friday March 06, 2015 @08:20PM (#9160561)
It's not trivial, but it is fairly easy to beat S3 on storage costs alone, let alone the other factors.

I personally wouldn't use Backblaze's storage pod, so my numbers are based on equivalent components (and don't reflect what's possible if you negotiate prices and optimize for your particular needs -- Backblaze's costs are surely lower than this):

  11 Supermicro 36 drive 4U server : $4000 each
  1 48-port top-of-rack switch : $5000 each
  374 4TB drive : $400 each
  22 500GB drive : $200 each
  2 CDU : $3000 each
  Rack + integration : $10000 each
Total rack cost is $220,000 for 1.4PB or 700GB with basic 2-copy redundancy. Power draw is around 8kw, so 1 year of power is around $1100. Figure on a drive failure rate of 5%, memory 2%, PSU 3% and you might spend another $18,000 to stock spares. So besides your core datacenter costs, this solution will cost $239,100 to operate for a year. These costs only improve when you amortize the hardware over a 3 year period, which is standard.

Now how much does S3 cost to store 700GB for a year? $248,530.94.

Finally, consider bandwidth costs. I'll crib from a recent comment of mine [1]:

1 gbit/s in a datacenter: $30,000 / year 500 mbit/s in S3: $142,540 / year

And that in a nutshell is why AWS doesn't make financial sense at any significant scale. S3 makes sense if you aren't really going to use all of that space: you can scale your spending up as your needs scale up. That isn't the kind of problem Backblaze or other storage businesses have, there isn't much chance of that storage going unused for any extended period of time. And if they architect their datacenter and service correctly, the down period between rolling in hardware and getting it utilized sufficiently to begin seeing a return on the investment should be minimal.

[1] https://news.ycombinator.com/item?id=9139244

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by toomuchtodo on Friday March 06, 2015 @08:34PM (#9160594)
Thanks for doing the math. I've repeatedly shown people (I do Infrastructure, sometimes AWS sometimes physical) that AWS is awesome for proof of concept. All opex, no commitments. As soon as you start getting to a certain scale though, time to move onto physical equipment when you've got predictable load patterns.

If Stackoverflow can still do hardware colo'd, so can you.

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by skuhn on Friday March 06, 2015 @08:41PM (#9160613)
I completely agree. I sometimes take flack for "hating" AWS (and I do hate some aspects of it), but I think it's a great platform for incubating your service or proving a concept. Every company I've been at in the last five years has used AWS to an extent, despite operating their own datacenters. It's great for random analytics jobs or other tasks that aren't steady work tied to your core business.

It quickly starts to not make sense for services that have usage-driven growth patterns. AWS's billing model just doesn't work for this -- even though higher usage is discounted, it isn't enough to overcome the skew of the model. Companies that have deployed on AWS often realize this too late: their business starts to take off, and they get crushed under giant AWS bills that would be disproportionate even at 50% off list. It's much harder to move off once you've hit scale.

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by acdha on Friday March 06, 2015 @10:32PM (#9160880)
A lot of this also comes down to ops & the features you really need – e.g. your example above didn't include the costs of the geographic redundancy or strong data integrity features which are standard with S3. A lot of places simply don't have geographically redundant datacenters and either lack the support team to run something like a replicated cluster filesystem in heavy production usage or, more commonly, don't have enough usage to really push it into clear win.

In the case above, the cost differential at 700TB is roughly the cost of a single sysadmin and so the running operation would probably be a wash versus S3 unless it had heavy data churn to maximize the S3 expenses or a good way to amortize the staffing needed for 24-hour-a-day support across other projects. Many places have budget processes which favor spending more on known upfront costs than getting permission to add full-time employees in the hopes of being able to beat those costs down the road.

I'm hoping that something like OpenStack Swift matures to the point where some of the complexity at the software + ops level starts making the DIY option routine. You'll never get rid of the minimum sysadmin commitment but that becomes a lot easier if it can be 5% of the night-shift because the software doesn't require a lot of care and feeding.

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by skuhn on Saturday March 07, 2015 @12:06AM (#9161065)
Yeah, my solution is a gross oversimplification -- my math is just back-of-the-napkin stuff, you can actually do much better on component pricing (I certainly don't pay $5000 for a 1G top-of-rack switch) and there are other variables to consider.

To quickly address some of your points:

Geographic redundancy: not included. From a hardware perspective it's a pretty easy upgrade (build two datacenters in CA / VA, buy two racks, put one in each). From a service perspective, it's a bit more challenging. If storage is your business, these are the core components you should own though.

The US Standard region in S3 does sort of provide geographic redundancy, but it's a question of whether it's good enough for your use case. S3 is a complete black box, and you have no way of knowing with certainty that every data blob is actually in both geographic regions. Besides that, it's eventually consistent, so if you need absolute assurances that data is not lost after a write succeeds, you have to implement logic on top of S3 -- and that logic must run in EC2 instances, or you'll pay a fortune in bandwidth to execute it.

No other region in S3 provides geographic redundancy (or eventual consistency, the two are related), so if you wanted to deploy storage in Europe or APAC, you have to handle the geographic distribution yourself -- and it will cost double to store that data.

The small cost difference: 1 rack over 1 year has a pretty small differential between the two. But you can optimize the datacenter numbers, and the S3 numbers are what they are unless you get massive. For example, over 3 years the datacenter cost goes to around $275,000 whereas S3 is $745,000. That's a serious difference now. Ownership is a powerful advantage.

OpenStack Swift: I don't think it will ever be a good solution, based on the project's history. It's unfortunate, but hopefully something else will come along that does blob storage in a sane and scalable way. This is not rocket science, but there are some devils in those details. If a mature open source solution came along with immutable objects, erasure coding, background scrubbing and sane ops tools, it would blow Swift out of the water.

Finally, I think it's a (common) mistake to think that you can get by without ops personnel if you deploy on AWS. The actual site / dc ops component is pretty minor if you build right. Either way, someone still has to run the service, do capacity / project / budget plans, and other things. And with AWS you need people who are capable of winning arguments with their support -- where you will burn substantial time proving where fault lies.

S3 does a lot for you, but it's still a service that you build yours on top of and it's far from a perfect solution that never breaks. Buckets have to be primed before heavy usage. Your file naming scheme will impact your performance at scale. And so on. You need people to look at this, whether you call them ops engineers or sysadmins or developers.

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by acdha on Saturday March 07, 2015 @10:19AM (#9161955)
> Geographic redundancy: not included. … If storage is your business, these are the core components you should own though.

Definitely – part of what I was thinking about is that it's also a good point to actually ask what you actually need. My experience has been in fields (science, digital preservation) where it turns out that people need large amounts of storage and strong guarantees against data corruption but online access is less critical since almost everyone works with more manageable derivatives than the original data. Unlike, say, a database there also aren't concerns with multiple writers since each batch of data originates in one place and should rarely if ever change, even though multiple locations might be generating other batches at the same time which everyone would want to read. That offers a ton of optimizations which are much harder to do if you're trying to maintain the same public contract as S3.

> OpenStack Swift: I don't think it will ever be a good solution, based on the project's history

Sadly, I've been coming to the same conclusion. I've been meaning to look at GlusterFS again as they've been adding things like erasure coding & the API portion of Swift but last time I checked they still haven't added strong integrity checks or scrubbing.

> Finally, I think it's a (common) mistake to think that you can get by without ops personnel if you deploy on AWS. The actual site / dc ops component is pretty minor if you build right. Either way, someone still has to run the service, do capacity / project / budget plans, and other things.

Agreed. The main thing I was referring to is that if you have to run a 24x7 service you have certain minimum staffing requirements. That's easily justified if you have enough demand but a small operation even inside a large company might find it easier to let AWS handle the basic ops so their staff can handle the higher-level ops work during normal business hours. (This is again a good opportunity to review the actual business needs to ask whether you really need immediate responses at 3am or can live with a delay while someone gets paged)

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by vidarh on Saturday March 07, 2015 @07:52AM (#9161719)
It's funny how people thing AWS does away with ops. But doing ops work for people on AWS is highly lucrative - people have a really hard time getting it right.
Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by viraptor on Saturday March 07, 2015 @07:14AM (#9161663)
I'm curious - since you went through trying out swift, when was that? Pre, or post diablo? That was the time of a pretty big change.

Also, have you tried ceph at the same scale?

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by skuhn on Saturday March 07, 2015 @05:47PM (#9163397)
Post-Diablo, and it never got to a point where I deployed it at scale and found issues: it flunked out before that.

Haven't tried ceph, although I look at it every few years. It sounds promising but it also sounds really complicated to setup and administer.

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by toomuchtodo on Saturday March 07, 2015 @01:17AM (#9161207)
It seems we've traveled the same path at different times :)
Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by parasubvert on Saturday March 07, 2015 @09:35AM (#9161875)
So, why do Instagram and Netflix continue to use S3 for petabytes of video and image data? In Netflix' case, this doesn't even count their vast replicated CDN infrastructure between your device and S3.

The math is never this simple.

Firstly I assume you mean 700 TB, not GB.

You might as well look at reduced-redundancy S3 which brings the annual cost down to $194,220. With that you still have multiple WAN links, excellent throughput, multiple data centers, and no labour for dealing with failed drives.

And if we're truly trying to do backup / archival, where download throughput doesn't matter, let's look at Glacier, even if we do 50000 monthly uploads and retrieve 20% of the 700TB monthly, we're down to $96,630.

Most sizable businesses also aren't buying bargain basement drive arrays from SuperMicro, they're buying EMC, NetApp, 3PAR, or Hitachi.

As with anything S3 is not perfect, but it really depends on your use case if your own data center or rented equipment is appropriate. For many large businesses, Amazon makes more sense.

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by skuhn on Saturday March 07, 2015 @05:51PM (#9163408)
> In Netflix' case, this doesn't even count their vast replicated CDN infrastructure between your device and S3.

There is part of your answer. They only pay S3 outbound on cache misses, not on cache hits. That will lower their S3 bandwidth bill to 1/50th or 1/100th of the full cost.

Also, Netflix doesn't pay list prices for AWS, and they made a call that they wanted to be fully cloud based early in the streaming service's life. Maybe they would do it differently today, but that ship has obviously sailed for them.

On the other hand, lots of large AWS hosted services do discover that they need to move off in order to get their costs right; I have quite a bit of first hand experience there.

> Glacier

Costs and time to retrieve data from Glacier are quite different, and I would only use it for data that I basically never expect to access. A service like Backblaze couldn't use Glacier in reality, despite being a service for backups.

> Bargain basement drive arrays

On the contrary, I think that building it yourself with basic hardware building blocks is much more common at large scale Internet companies. A less technical or more appliance driven company might choose NetApp or EMC, but I don't think it's the right move for somewhere with some in-house expertise. It's the right move if you want more of a black box that you hope will work without much internal effort (hint: it won't work out that way).

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by moe on Saturday March 07, 2015 @04:41AM (#9161458)
Your cost estimates are outlandish, making Amazon S3 look like a bargain.

In reality 1.4PB raw will set you back well below $50k/pa, not $239k/pa.

If you're lazy you could even rent 1.4PB raw from e.g. Hetzner and you'd still be paying under $90k/pa (that would be 23 of their SX290 servers).

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by skuhn on Saturday March 07, 2015 @05:26AM (#9161527)
You can do significantly better with the right vendors and some negotiation -- this is specifically a pricing ballpark, not rock bottom deals. It's not outlandish at all to say that in terms of raw storage you're going to be close to S3's pricing, but it also doesn't reflect well on S3 at all. They are buying at a scale far beyond what you can hope to achieve.

This is money spent on physical equipment that you can choose to lease, that you can take depreciation on, that you can resell -- these are assets, not a line item on a monthly bill. S3 has no such benefits. And this doesn't even taking into account bandwidth, which is the real killer. The stored data has to come out sometime.

1.4PB of raw storage requires ~374 4TB hard drives. If you want to spend $50,000, you have at most $130 to spend on drives, and that's only if you get literally everything else for free including electricity. You can almost buy 4TB desktop class drives for that price from Newegg, but not quite.

Don't buy drives from retail channels, and don't buy desktop drives at all (yes, I know Backblaze does both of these things). HGST 4TB MegaScale or UltraStar drives are a safe bet, depending on your performance requirements. They run between $250-300 in quantity, which means 1.4PB is $93,500 in disks alone.

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by moe on Saturday March 07, 2015 @08:24AM (#9161765)
Stop switching around between hardware cost and yearly cost, jesus... (pa means per annum)

0.7PB on Amazon S3 costs $810k for 3 years (as per their cost calculator).

Renting 1.4PB raw from Hetzner costs about $270k for 3 years.

Running 1.4PB raw on your own hardware costs about $150k for 3 years (on Supermicro JBODs).

It's not that hard really, and my point merely was that the difference is much bigger than you made it sound.

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by parasubvert on Saturday March 07, 2015 @09:54AM (#9161908)
Apples, to apples please.

0.7PB on Amazon S3 reduced redundancy is $582,660 over three years with geo-redundancy and availability across an entire region, and lifecycle to push to Glacier which would drop the price to $289,872 annually given 50,000 monthly uploads and 20% monthly retrieval. No labour when it comes to dealing with equipment failures.

I'd also say that only SMBs and web/internet companies run the equivalent SuperMicro JBODs, most large companies use enterprise storage vendors like EMC, NetApp, HP/3PAR which are premium priced.

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by moe on Saturday March 07, 2015 @10:23AM (#9161965)
Apples, to apples please.

You are right, I should have compared to S3 reduced redundancy, sorry.

So we're down to $582k (S3-RR) vs $270k (Hetzner) vs $150k (own hardware).

That's still a healthy 2x-4x difference.

Glacier

If you introduce Glacier then it's not apples-to-apples anymore.

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by parasubvert on Saturday March 07, 2015 @01:17PM (#9162527)
"If you introduce Glacier then it's not apples-to-apples anymore."

This is a thread about storage costs for backup as a service, so I felt at least it's in the same family of tree-bearing fruit ;)

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by vidarh on Saturday March 07, 2015 @07:47AM (#9161711)
I've done similar calculations for clients, and fully agree.

Cutting the storage costs is easy, but bandwidth in particular is ridiculously priced at Amazon (for the cost of 1TB transfer from AWS, I can rent a server with many times as much transfer and still have money left over), to the point where even if a client for some reason insists on S3 storage for perceived safety, it can still pay to store in S3 but put boxes "in front" with capacity for a single copy of each object, and fall back to S3 in case of local drive failures.

Unless the load is totally write dominated (so not good for backup services), cutting the AWS bandwidth bill can often pay for the extra servers and colo costs many times over.

Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by jrs235 on Friday March 06, 2015 @11:03PM (#9160943)
I think you mean 700TB not 700GB
Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by skuhn on Friday March 06, 2015 @11:44PM (#9161029)
Right, just a unit problem -- the pricing is still correct. I seem to make this mistake a lot lately, since everything is billed per-gig or per-meg, but you only really care about per-TB units. Oops.
Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by atYevP on Saturday March 07, 2015 @12:41AM (#9161128)
Don't you mean...ops? Hah...sorry puns :-/
Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by nullc on Saturday March 07, 2015 @02:05AM (#9161278)
>1.4PB or 700GB with basic 2-copy redundancy

You mean 700TB.