Building a 30 PB storage cluster in the heart of SF

Re:Building a 30 PB storage cluster in the heart of SF by OutOfHere on Wednesday October 01, 2025 @12:07PM (#45439398)

Is it correct that you have zero data redundancy? This may work for you if you're just hoarding videos from YouTube, but not for most people who require an assurance that their data is safe. Even for you, it may hurt proper benchmarking, reproducibility, and multi-iteration training if the parent source disappears.

[ Parent | on HN ]

Re:Building a 30 PB storage cluster in the heart of SF by nee1r on Wednesday October 01, 2025 @12:08PM (#45439419)

Definitely much less redundancy, this was definitely a tradeoff we made for pretraining data and cost.

[ Parent | on HN ]

Re:Building a 30 PB storage cluster in the heart of SF by Sanzig on Wednesday October 01, 2025 @12:36PM (#45439797)

Did you do any kind of redundancy at least (eg: putting every 10 disks in RAID 5 or RAID Z1)? Or I suppose your training application doesn't mind if you shed a few terabytes of data every so often?

[ Parent | on HN ]

Re:Building a 30 PB storage cluster in the heart of SF by g413n on Wednesday October 01, 2025 @01:36PM (#45440564)

atm we don't and we're a bit unsure whether it's a free lunch wrt adding complexity. there's a really nice property of having isolated hard drives where you can take any individual one and `sudo mount` it and you have a nice chunk of training data, and that's something anyone can feel comfortable touching without any onboarding to some software stack

[ Parent | on HN ]

Re:Building a 30 PB storage cluster in the heart of SF by randomfinn on Wednesday October 01, 2025 @08:51PM (#45445325)

I wonder if snapraid would work for this. Especially if your data is mostly written once and then just read, it could be an easy way to add redundancy while keeping isolated individual drives.

[ Parent | on HN ]