News for hackers. Stuff that matters.
Re:Ask HN: How would you store 10PB of data for your startup... by Tepix on Friday April 23, 2021 @12:04PM (#26916151)
It seems to me like you could save a ton of money by using your own hardware. Perhaps buy a bunch of big Synology boxes? At that scale you should also consider looking at technologies such as Ceph.

We've recently switched to a setup with several Synology boxes for around 1PB net storage.

Re:Ask HN: How would you store 10PB of data for your startup... by faeyanpiraat on Friday April 23, 2021 @05:36PM (#26919494)
Those boxes are slooooow, the 8 slot box have like a 500MB/s read speed limit, even if you raid0 8 SSDs, and use 10gbps networking. This limitation is in the product spec document, but with the smallest letters possible.
Re:Ask HN: How would you store 10PB of data for your startup... by Tepix on Wednesday April 28, 2021 @05:51AM (#26967593)
They advertise some boxes with 5.5GB/s.
Re:Ask HN: How would you store 10PB of data for your startup... by nobozo on Friday April 23, 2021 @01:43PM (#26917211)
Funny you should mention this. I once worked at a startup that stored lots of remote sensing data. Their strategy was to put it on a Synology. When the Synology filled up, they bought another, and so forth. Only some of the Synologys were online at any particular time, and there was no indexing to find which Synology held what data.

Plus, there were no backups so if one Synology were to blow up, all the data on it was lost.

Since they were a small startup it made some sense to start this way, but they had no plans on what to do about it as they got bigger.

Re:Ask HN: How would you store 10PB of data for your startup... by Tepix on Friday April 23, 2021 @04:06PM (#26918637)
Using Synologys doesn't mean that you have to be stupid about it :-)
Re:Ask HN: How would you store 10PB of data for your startup... by philippb on Friday April 23, 2021 @01:05PM (#26916793)
Thank you!