Re:Etsy is under pressure to start acting more like a conven...
by smileysteve on Thursday May 18, 2017 @04:14PM
(#14370710)
You still need software ops, yes, but not hardware ops.
> Hardware failures are far less common than you believe.
Your experience may vary; I once rented metal from Softlayer (running ~15 shards, about 60 boxes), we had a number of drives* fail, a couple of the rack controllers, some raid controllers, and one time a power supply over a 3 year period. On the worst RAID failure, we sent one of our employees across country to manage the recovery directly.
*Some hard drive failures related to 2012 Seagate 3TB drive issue. One failed within a week of being replaced.
Softlayer had a team monitoring our servers and working through issues with us; Other than supplier issues I blame them for nothing.
In that worst scenario; we ran off our geo redundant slave for the better part of a week.
Re:Etsy is under pressure to start acting more like a conven...
by nasalgoat on Thursday May 18, 2017 @04:23PM
(#14370775)
I think you overstate the overhead for hardware operations. I had a team of five that managed over 1,000 physical servers in four global locations and they had to visit a datacentre maybe twice in three years. Other than for hardware installs which were always easily planned out.
Hardware failures are far less common than you believe. I lost one hard drive in 10 years and because it was a RAID array it didn't even fail.