Re:Building a 30 PB storage cluster in the heart of SF
by trebligdivad on Wednesday October 01, 2025 @04:54PM
(#45443386)
I assume your actual training is being done somewhere else? Did you try getting colocation space in the same datacentre as somewhere with the compute - it would have reduced your internet costs even further.
[ Parent
| on HN ]
Re:Building a 30 PB storage cluster in the heart of SF
by g413n on Wednesday October 01, 2025 @05:00PM
(#45443461)
yeah the cost calculus is very different for gpus, it absolutely makes sense for us to be using cloud there. also hardly any datacenters can support the power density, esp in downtown sf
Re:Building a 30 PB storage cluster in the heart of SF
by trebligdivad on Wednesday October 01, 2025 @05:50PM
(#45443960)
Yeh; one other thing - you list a separate management network as an optional - it's not optional! Under no circumstance must you expose the managemnt IPs of switches or the servers to the internet; they are, on average, about as secure as a drunk politician. Use a separate management net, make sure it's only securly accessed.
Re:Building a 30 PB storage cluster in the heart of SF
by Symbiote on Wednesday October 01, 2025 @06:37PM
(#45444436)
I understood that it's optional because they can walk down the road to the data center instead.
They mention plugging monitors in several times. I think I've only done that once in the last couple of years, when a firmware upgrade failed and reset the management interface IP.