News for hackers. Stuff that matters.
Re:Reddit was down for over two hours by paxys on Wednesday September 18, 2019 @06:52PM (#21011302)
A website is not a subway system. There are no "huge costs" to having basic rolling deployments set up. In fact it's likely more expensive to have extended outages and maintenance windows, because it shows that there are humans running commands behind the scenes rather than automation.
Re:Reddit was down for over two hours by stevenpetryk on Wednesday September 18, 2019 @07:39PM (#21011694)
Just wondering (sort of—I'm also slightly trying to point out that you're being a bit of an armchair Reddit ops person :)

What makes you think that the issue Reddit faced today would be fixed by rolling deploys?

Re:Reddit was down for over two hours by paxys on Wednesday September 18, 2019 @07:45PM (#21011745)
I definitely am (but we are on Hacker News after all).

Without knowing anything about this particular issue, at a basic level I'd think any non-trivial site should have a system of rolling out changes to a small percentage of servers and have automated monitoring that fires alerts if it notices any disruptions in functionality. If nothing goes wrong then gradually roll out to 100%. If something does then stop the rollout and go back to the last stable version. Then engineers can look at what went wrong.

Re:Reddit was down for over two hours by hobofan on Thursday September 19, 2019 @03:10AM (#21014009)
From my experience that mostly only applies to the application layer. Especially updates on distributed databases are often not as easy as "stop and rollback" as you have to achieve a stable quorum or other similar dynamic states like that which you can't easily achieve from restoring a snapshot. There your best shot often is to learn how to deal with all the failure scenarios manually and do as many tests on a staging system until you feel comfortable.