A Google Cloud support engineer solves a tough DNS case

Re:A Google Cloud support engineer solves a tough DNS case by dekhn on Tuesday May 19, 2020 @04:10PM (#23239560)

Once, the head of IT of a company I used to work for was touring the datacenter, passing some new racks filled with blade servers. He stopped, said "why are all the fans running full blast on this rack?" and the admins checked and they were running some test workload at scale somebody had forgotten about a few weeks before.

Everybody was embarassed because no monitoring caught it, but the VP of IT did by walking past the cage.

[ Parent | on HN ]

Re:A Google Cloud support engineer solves a tough DNS case by killjoywashere on Tuesday May 19, 2020 @10:47PM (#23242800)

Admiral Rickover was known for walking into the engineering spaces of nuclear ships and just throwing a valve handle that would force a reactor scram. Not infrequently on a submerged submarine. Just to make sure the team was on their toes.

[ Parent | on HN ]

Re:A Google Cloud support engineer solves a tough DNS case by MaxBarraclough on Wednesday May 20, 2020 @05:04AM (#23244787)

I'm not a sailor or a nuclear engineer, but that doesn't sound like a great idea. Should the Chaos Monkey approach really be used on nuclear systems?

Aside: was it even legal for him to do that?

[ Parent | on HN ]

Re:A Google Cloud support engineer solves a tough DNS case by killjoywashere on Wednesday May 20, 2020 @05:11AM (#23244816)

If you are responsible for building the industry that designs and builds nuclear submarines that carry nuclear missiles, you had better make sure that those submarines and their crews can handle chaos monkeys.

Also, Rickover was Congress's favorite admiral. They forced the Navy to promote him. I'm pretty sure they made sure that the laws were to his liking.

[ Parent | on HN ]

Re:A Google Cloud support engineer solves a tough DNS case by dekhn on Wednesday May 20, 2020 @12:32PM (#23248843)

Rickover was the sort of person who had the technical expertise, gumption, and charisma to get away with this. It's worth reading up on this amazing individual, who made navy's nuclear reactors so safe, and led the creation of nuclear reactor expertise within the navy.

Also, if you're not confident enough in your nuclear reactor to apply chaos monkey techniques, you shouldn't be engineering nuclear reactors.

[ Parent | on HN ]

Re:A Google Cloud support engineer solves a tough DNS case by nexuist on Wednesday May 20, 2020 @08:07AM (#23245869)

In general, one would hope that a nuclear system is designed such that the problem can easily be corrected if a single button or lever is accidentally pressed. It would be quite a terrible system if you could e.g. trigger a meltdown with just one action.

[ Parent | on HN ]

Re:A Google Cloud support engineer solves a tough DNS case by bovermyer on Wednesday May 20, 2020 @08:55AM (#23246292)

A reactor scram is basically just an emergency shutdown. If I were a nuclear engineer, it might give me a heart attack to hear the scram alarms, but I would be plenty happy knowing the scram works.

[ Parent | on HN ]

Re:A Google Cloud support engineer solves a tough DNS case by gerdesj on Tuesday May 19, 2020 @06:30PM (#23241061)

I walk the server room daily, every morning. I've tended our monitoring system for 15 years now and I don't trust myself to be infallible. I'm also the MD ...

[ Parent | on HN ]

Re:A Google Cloud support engineer solves a tough DNS case by jiveturkey on Wednesday May 20, 2020 @04:49PM (#23251627)

MD = VP for those not in finance

[ Parent | on HN ]

Re:A Google Cloud support engineer solves a tough DNS case by bennyelv on Thursday May 21, 2020 @03:29AM (#23255748)

In the UK MD = CEO (when not finance), so without more context you couldn’t really say.

He’s either an MD in finance, which is the equivalent of a VP (roughly) in other kinds of companies, or he’s the CEO.

[ Parent | on HN ]