The amount of downtime this would generate for a single machine plus the operation cost doesn't worth the hassle unless the machine loses a significant chunk of drives.
Also you've clearly never interacted with cattle, sheep, goats, llamas or alpacas, which absolutely do get things like veterinary care and vaccinations. Large animal vet is a whole specialty and they spend lots of time working on animals other than horses. No trips to the vet???
For what it’s worth, meat packers buy dead cattle and don’t ask questions. But this is a well known metaphor, and I’m pointing out by that metaphor, no trips to the vet. As a cattle farmer, I’d argue it holds true if you’re big enough they’ve got tags not names: the vet comes to you and only if you think you’ve got a herd problem instead of an individual problem.
As for the HN angle: these are contrarian and objection-inspiring policies that let us wholesale video delivery to/through other CDNs while making a profit.
Farmers absolutely have a vet who takes care of the cattle. I am not sure what you are on about.
The whole point of cattle-vs-pets is you are supposed to treat all the servers the same, not that you have to treat all of them poorly.
Another contrarian view — particuarly suitable for large VDN content like media, not small CDN content like html/js — is that one doesn’t need to be in over a hundred data centers, one needs to be in the key exchanges: you don’t have to be at the ends of every spoke if you pick the right hubs.
Agree swapping bad drives is a reasonable use of smart hands when done in batches, as no diagnosis is needed. I’d advocate considering extending that practice to the servers themselves. Math works if you find local tech repo/refurb shops that take gear (w/o drives) to bulk refurb & resell. The other way is to get your OEM to provide aliveness-as-a-service.
Anything so you don’t have to do manual labor, ideally ever.
I think at that scale you would want a ceph expert on staff as a full time salaried position.
For an organization that has 10PB now and can project a growth path to 15, 20, 25PB in the future, you should talk with management about creating a vacant position for that role, and filling it.
> EDIT: btw, we let the dead drives "rot": replacing them would cost more, and the failure rate is not that bad, so they stay in the machine, and we disable them in fstabs, configs, etc.
I am a huge advocate of hosting stuff yourself on bare metal you own, but this is a ridiculous statement. Any drive in that class should come with a 3 or 5 year warranty. And the manual labor and hassle time to replace one (you have hundreds of thousands of dollars of storage and no ready to go cold spares on a shelf?!?!) is infinitesimal.