News for hackers. Stuff that matters.
Re:Migrating 23TB from S3 to B2 in 7 hours by dgemm on Friday April 12, 2019 @10:56PM (#19650819)
If this is purely backup data, wouldn't glacier be a better fit than either S3 or B2?

Glacier is already cheaper than B2 and has the advantage of storing data redundantly across datacenters. And glacier deep archive is 4 times cheaper than that.

Full disclosure: AWS employee

Re:Migrating 23TB from S3 to B2 in 7 hours by CherryJimbo on Friday April 12, 2019 @11:10PM (#19650872)
They're "backups" in the sense of storing customer data, but realistically they're more like instance snapshots for various customer game servers. They're created and restored multiple times a day for every customer as they hop between games (accessed very frequently), so Glacier wouldn't be a good fit.
Re:Migrating 23TB from S3 to B2 in 7 hours by hrez on Sunday April 14, 2019 @03:21AM (#19658034)
If it's that much churn, wouldn't it make sense to point new writes/updates to B2 and let s3 naturally drain/lifecycle out its data?
Re:Migrating 23TB from S3 to B2 in 7 hours by dgemm on Sunday April 14, 2019 @02:38PM (#19660906)
Probably a technically better solution but it would not make as good of a PR piece as an epic migration.
Re:Migrating 23TB from S3 to B2 in 7 hours by amenod on Sunday April 14, 2019 @03:17PM (#19661130)
Actually I think it would sound better - "How to move 23TB without moving (almost) anything". Especially to technical people.
Re:Migrating 23TB from S3 to B2 in 7 hours by Twirrim on Saturday April 13, 2019 @12:59AM (#19651193)
B2 didn't used to be multiple ADs, or at least not multiple ADs over a wider geographic area, like AWS's regions. I'm not sure if that's changed at all. It used to be why B2 was able to be cheaper. You were paying for less durability and availability.

S3 storage is redundant over an entire region. Entire availability zones can go down (extremely improbable) without your data being put at risk.

It, therefore, struck me as odd to see:

> Due to S3 and B2 being at least nearly equally* accessible, reliable, available, as well as many other providers, our primary reason for moving our backups now became pricing.

but then ...

> * Science is hard, blue keys on calculators are tricky, and we don’t have years to study things before doing them

WTF? They're trying to be joking here I hope, but that comes across somewhat "We don't give a crap about our customer's data".

Re:Migrating 23TB from S3 to B2 in 7 hours by gregorysudderth on Saturday April 13, 2019 @04:20AM (#19651739)
Sorry for the bad vibe.

I was trying to interject some humor into a dry process. When I asked a couple peers from my storage background, "how would you do this project" they both said the same thing: 1) form a working group 2) do a study 3) test 4) 90 days later, do an analysis. Epic process.

We care a lot, but that's beyond our scope. This was an all-hands reviewed process, and like machines that are built to be ultimately reliable including: hospital generators, human-rated spacecraft equipment, airplane engines for single-engine remote location flying (PT6), our process was detuned speed-wise specifically, for overall service reliability and minimal impact.

We tested, we analyzed, we saw weird failures (python core dump?) in simple code, and followed up on all of those before proceeding. We agonized.

In the end we were satisfied, because our customers couldn't tell anything changed, and our CSR's got zero new tickets. Not bragging at all, relieved of course, but, we care about the CSR's load and work experience too.

Thanks for checking the article out, and let me know if we can show you more about our service.

Re:Migrating 23TB from S3 to B2 in 7 hours by snissn on Friday April 12, 2019 @11:06PM (#19650854)
No - the pricing on glacier makes it a steaming pile of shit
Re:Migrating 23TB from S3 to B2 in 7 hours by camtarn on Saturday April 13, 2019 @02:35AM (#19651433)
I was pleasantly surprised to see that the previous rather crazy retrieval pricing model (price based on the peak retrieval bandwidth you used at any time in the current month) was replaced by a straight price-per-GB model a couple years back. That made my life a lot easier when pricing up Glacier vs B2 for a client.
Re:Migrating 23TB from S3 to B2 in 7 hours by dgemm on Friday April 12, 2019 @11:09PM (#19650865)
It's built for off-site backup type workloads, where you write once and read hopefully never.

You are probably using it wrong.