Re:Etsy is under pressure to start acting more like a conven...
by nasalgoat on Thursday May 18, 2017 @04:23PM
(#14370775)
I think you overstate the overhead for hardware operations. I had a team of five that managed over 1,000 physical servers in four global locations and they had to visit a datacentre maybe twice in three years. Other than for hardware installs which were always easily planned out.
> Hardware failures are far less common than you believe.
Your experience may vary; I once rented metal from Softlayer (running ~15 shards, about 60 boxes), we had a number of drives* fail, a couple of the rack controllers, some raid controllers, and one time a power supply over a 3 year period. On the worst RAID failure, we sent one of our employees across country to manage the recovery directly.
*Some hard drive failures related to 2012 Seagate 3TB drive issue. One failed within a week of being replaced.
Softlayer had a team monitoring our servers and working through issues with us; Other than supplier issues I blame them for nothing.
In that worst scenario; we ran off our geo redundant slave for the better part of a week.