News for hackers. Stuff that matters.
Re:Ask HN: How would you store 10PB of data for your startup... by tw04 on Friday April 23, 2021 @03:44PM (#26918405)
I should preface this with: I read the question as you want something on-premises/in a colo. If you're talking hosted S3 by someone other than Amazon that's a different story.

It probably depends on if you are tied at the hip to other AWS services. If you are, then you're kind of stuck. The ingress/egress traffic will kill you doing anything with that data anywhere else.

If you aren't, the major players for on-prem S3 (assuming you want to continue access the data that way) would be (in no specific order):

Cloudian

Scality

NetApp Storagegrid

Hitachi Vantara HCP

Dell/EMC ECS

There are plusses and minuses to all of them. At that capacity I would honestly avoid a roll-your-own unless you're on a shoestring budget. Any of the above will be cheaper than Amazon.

Re:Ask HN: How would you store 10PB of data for your startup... by qt_scientist on Saturday April 24, 2021 @01:24AM (#26922479)
[Disclaimer: I work on Quantum ActiveScale, an on-prem S3 system that fits this list]

Yup this is why these vendors exist. You’re definitely not alone, cloud repatriation is a ‘thing’.

These vendors have replaced that Ceph or Minio expert others in this thread said you’d need to budget for, by software. The system detects dead/degrading disks and automatically evicts them and rebuilds that chunk of the erasure code on another disk. Every few months you go in and hot swap the bad disks = the ones with a blinking led. Also Prometheus metrics, alerts in case of issues,... You don’t need a storage admin babysitting this.

1 rack of 4u90 enclosures with 18TB disks is 15PB RAW so with erasure coding overhead about the 10PB usable you need today.

I’m obviously biased on which vendor to pick. Do your due diligence, f.i. on how the system does capacity expansions.