News for hackers. Stuff that matters.
Re:Backblaze Storage Pod 4.5 – Tweaking a Proven Design by acdha on Saturday March 07, 2015 @10:19AM (#9161955)
> Geographic redundancy: not included. … If storage is your business, these are the core components you should own though.

Definitely – part of what I was thinking about is that it's also a good point to actually ask what you actually need. My experience has been in fields (science, digital preservation) where it turns out that people need large amounts of storage and strong guarantees against data corruption but online access is less critical since almost everyone works with more manageable derivatives than the original data. Unlike, say, a database there also aren't concerns with multiple writers since each batch of data originates in one place and should rarely if ever change, even though multiple locations might be generating other batches at the same time which everyone would want to read. That offers a ton of optimizations which are much harder to do if you're trying to maintain the same public contract as S3.

> OpenStack Swift: I don't think it will ever be a good solution, based on the project's history

Sadly, I've been coming to the same conclusion. I've been meaning to look at GlusterFS again as they've been adding things like erasure coding & the API portion of Swift but last time I checked they still haven't added strong integrity checks or scrubbing.

> Finally, I think it's a (common) mistake to think that you can get by without ops personnel if you deploy on AWS. The actual site / dc ops component is pretty minor if you build right. Either way, someone still has to run the service, do capacity / project / budget plans, and other things.

Agreed. The main thing I was referring to is that if you have to run a 24x7 service you have certain minimum staffing requirements. That's easily justified if you have enough demand but a small operation even inside a large company might find it easier to let AWS handle the basic ops so their staff can handle the higher-level ops work during normal business hours. (This is again a good opportunity to review the actual business needs to ask whether you really need immediate responses at 3am or can live with a delay while someone gets paged)

There are no comments on this comment yet.