News for hackers. Stuff that matters.
Re:Managed services killed DevOps by toomuchtodo on Friday April 08, 2016 @11:08AM (#11455151)
I'm a devops engineer, and I assure you, I have been on call for the last two years in my current position, attending to application and infrastructure alarms.

All disaster recovery cannot be automated. A sanity check is required before your Elasticsearch or Redis clusters decide to f___ their datastore.

EDIT: Or perhaps RDS is to your liking? We've had our replicas (off the same master) in AWS fail, at which point they would not further replicate. We needed to fail our app over to the master entirely, reduce its workload to not kill the master, terminate all of the replicas, relaunch all of the replicas, and then re-apply the workload back to the new replicas.

Re:Managed services killed DevOps by creshal on Friday April 08, 2016 @11:12AM (#11455189)
> All disaster recovery cannot be automated.

See also: https://github.com/blog/1261-github-availability-this-week

> The automated failover of our main production database could be described as the root cause of both of these downtime events. In each situation in which that occurred, if any member of our operations team had been asked if the failover should have been performed, the answer would have been a resounding no. There are many situations in which automated failover is an excellent strategy for ensuring the availability of a service. After careful consideration, we've determined that ensuring the availability of our primary production database is not one of these situations. To this end, we've made changes to our Pacemaker configuration to ensure failover of the 'active' database role will only occur when initiated by a member of our operations team.

Re:Managed services killed DevOps by toomuchtodo on Friday April 08, 2016 @11:14AM (#11455210)
Pinterest as well uses manual intervention:

https://engineering.pinterest.com/blog/open-sourcing-pintere...

Re:Managed services killed DevOps by lisivka on Friday April 08, 2016 @11:29AM (#11455350)
All disaster recovery in general case cannot be automated, you are right. But it definitely can be automated to the point, when problem can wait till next Monday.

To recover any disaster automagically, you need five things: your actual data, your working software, a working hardware, a working network connection, money. All that, except actual data, can be prepared ahead of time.

Re:Managed services killed DevOps by toomuchtodo on Friday April 08, 2016 @11:33AM (#11455379)
If your datastore can run in a degraded state through a weekend, its not a critical datastore (or you're shedding writes or making other consistency compromises).
Re:Managed services killed DevOps by lisivka on Friday April 08, 2016 @12:37PM (#11455957)
If you have a single point of failure, then it is your fault.