Building a 30 PB storage cluster in the heart of SF

Re:Building a 30 PB storage cluster in the heart of SF by rtp4me on Thursday October 02, 2025 @06:59AM (#45448161)

For drive issues, this is easy. Have a stack of replacements on hand and just open a "remote-hands" ticket with the CoLo provider to swap out the drive. This can usually be done in 1-2hrs from opening the ticket.

For server issues; again, pretty easy. Just use iKVM/IPMI and iPXE to diagnose a faulty server. Again, using "remote-hands" from the CoLo provider can help fix problems if your staff does not have the skills.

[ Parent | on HN ]

Re:Building a 30 PB storage cluster in the heart of SF by Aurornis on Thursday October 02, 2025 @09:16AM (#45449224)

In my experience, the issues that take 80% of your time are the unexpected edge cases, not the easy fixes.

Swapping drives is basically the easiest fix. The issues that cause the most problems are the hard to diagnose ones like the faulty RAM that flips a bit every once in a while or the hard drive controller that triggers an driver bug with weird behavior that doesn’t show up in the logs with anything meaningful.

[ Parent | on HN ]

Re:Building a 30 PB storage cluster in the heart of SF by rtp4me on Thursday October 02, 2025 @10:12AM (#45449845)

Sure, but realistically, how often does this really happen? I have probably replaced 3 or 4 DIMMs over the past few years. Hardware is very reliable these days.

[ Parent | on HN ]