Re:Building a 30 PB storage cluster in the heart of SF
by Sanzig on Wednesday October 01, 2025 @12:36PM
(#45439797)
Did you do any kind of redundancy at least (eg: putting every 10 disks in RAID 5 or RAID Z1)? Or I suppose your training application doesn't mind if you shed a few terabytes of data every so often?
[ Parent
| on HN ]
Re:Building a 30 PB storage cluster in the heart of SF
by g413n on Wednesday October 01, 2025 @01:36PM
(#45440564)
atm we don't and we're a bit unsure whether it's a free lunch wrt adding complexity. there's a really nice property of having isolated hard drives where you can take any individual one and `sudo mount` it and you have a nice chunk of training data, and that's something anyone can feel comfortable touching without any onboarding to some software stack
Re:Building a 30 PB storage cluster in the heart of SF
by randomfinn on Wednesday October 01, 2025 @08:51PM
(#45445325)
I wonder if snapraid would work for this. Especially if your data is mostly written once and then just read, it could be an easy way to add redundancy while keeping isolated individual drives.