News for hackers. Stuff that matters.
Re:Building a 30 PB storage cluster in the heart of SF by nee1r on Wednesday October 01, 2025 @11:36AM (#45439011)
True, though this is specifically for pretraining data (S3 wouldn't sell us used disk + no DR storage).
Re:Building a 30 PB storage cluster in the heart of SF by p_ing on Wednesday October 01, 2025 @11:54AM (#45439244)
You're in a seismically active part of the world. Will the venture last in a total loss scenario?
Re:Building a 30 PB storage cluster in the heart of SF by antonkochubey on Wednesday October 01, 2025 @12:12PM (#45439477)
They spent $300,000 on drives, with AWS they would have spent 4x that PER MONTH. They're already ahead of the cloud.
Re:Building a 30 PB storage cluster in the heart of SF by p_ing on Wednesday October 01, 2025 @02:43PM (#45441553)
AWS/cloud doesn't factor into my question what so ever. Loss of equipment is one thing. Loss of all data is quite a different story.
Re:Building a 30 PB storage cluster in the heart of SF by nee1r on Wednesday October 01, 2025 @12:00PM (#45439315)
We're currently 1/1 for the recent 4.3 magnitude earthquake (though if SF crumbles we might lose data)
Re:Building a 30 PB storage cluster in the heart of SF by p_ing on Wednesday October 01, 2025 @02:44PM (#45441568)
4.3 is a baby quake. I'd hope that you'd be 1/1!
Re:Building a 30 PB storage cluster in the heart of SF by Sanzig on Wednesday October 01, 2025 @12:44PM (#45439908)
I do appreciate the scrappiness of your solution. Used drives for a storage cluster is like /r/homelab on steroids. And since it's pretraining data, I suppose data integrity isn't critical.

Most venture-backed startups would have just paid the AWS or Cloudflare tax. I certainly hope your VCs appreciate how efficient you are being with their capital :)

Re:Building a 30 PB storage cluster in the heart of SF by g413n on Wednesday October 01, 2025 @01:39PM (#45440603)
worth stressing that we literally could not afford pretraining without this, approx our entire seed round would go into cloud storage costs