News for hackers. Stuff that matters.
Re:Building a 30 PB storage cluster in the heart of SF by cornholio on Thursday October 02, 2025 @07:04AM (#45448195)
30PB / 100Gbps comes down to about a month, 4 links would give you a week, so that seems pretty quite acceptable for a training run, especially since you can overlap the initial loading of the array with the first training, i.e train as data becomes available.

It goes without saying any data pre-processing needs to be done before writing, at the storage site, or on the training GPUs.

There are no comments on this comment yet.