News for hackers. Stuff that matters.
Re:Site Reliability Engineering by tehlike on Saturday January 28, 2017 @02:23AM (#13505350)
One of the cases i was involved was when the issue was not found after 30 minutes, after sre rolled back most of the systems.

Reproducing the issue resulted in an immediate fix by the swe.

Again, i understand why it is the way it is, it is just really interesting to see how specialized each engineer is in the grand scheme of things.

Re:Site Reliability Engineering by general_ai on Saturday January 28, 2017 @03:39AM (#13505519)
Immediate fix by SWE can only be released after it is tested and canaried, so it's not really "immediate" most of the time.
Re:Site Reliability Engineering by sokoloff on Saturday January 28, 2017 @11:21AM (#13507086)
We run factories and if we had a bad deploy bring a factory down, it's not going to get "more broken", so we can push a fix-attempt change live as soon as it's ready.

Abstractly, we got pushback from QA about this policy. After we had gathered a couple of concrete examples, it was clear that QA-as-gatekeeper when the factory is already in the worst possible state wasn't valuable. We do mandate the normal reviews but allow them after deployment. (You can imagine the conversations with the auditors about this as well, so we had to carefully document that this was our process and made the auditors audit our conformance to the process not to their own preconception of what it should be.)

Re:Site Reliability Engineering by tehlike on Saturday January 28, 2017 @12:09PM (#13507419)
That is not true. Production fires tend to skip canaries in a sizable amount of cases.

It's especially not true when big amount of money is at stake. (like ads).

Edit: last sentence.