News for hackers. Stuff that matters.
Re:Improving how we deploy GitHub by WJW on Monday January 25, 2021 @02:03PM (#25907213)
That seems like an extremely good idea actually, since if you dogfood your own releasing service then you can't fix it anymore if you accidentally bring down the service.
Re:Improving how we deploy GitHub by Xorlev on Monday January 25, 2021 @02:33PM (#25907641)
That also means when it does go wrong, it takes much longer to fix. Good operational practice is to decrease MTTR, not make it worse.
Re:Improving how we deploy GitHub by xxpor on Monday January 25, 2021 @02:33PM (#25907627)
That's usually solved with a parallel stack deployment, use the other stack if something is broken
Re:Improving how we deploy GitHub by paxys on Monday January 25, 2021 @02:34PM (#25907643)
If the "other stack" isn't regularly used then you can assume it will be broken when needed
Re:Improving how we deploy GitHub by cpascal on Monday January 25, 2021 @03:58PM (#25908777)
You just run the previous version of the production stack in your "dogfood/operations" stack. Once you've fully rolled out production and have vetted it, you can upgrade the other one to match production.
Re:Improving how we deploy GitHub by epidemian on Monday January 25, 2021 @08:44PM (#25911416)
With large codebases maintained by many people, sometimes it's difficult to "just" do things like that. It's a bit weird to think that no one within that large group of professional developers hasn't thought of doing a simple obvious solution. It's probably not that simple.
Re:Improving how we deploy GitHub by cutemonster on Tuesday January 26, 2021 @08:35PM (#25923214)
Lots of comments here around that say "just do it like this: ...", or "why don't you: ... it'd be good because ..."
Re:Improving how we deploy GitHub by marcinzm on Monday January 25, 2021 @10:00PM (#25911911)
And then you get hit with a subtle bug that utterly nukes your system when the year changes from 2020 to 2021. Now you can't deploy anything because both systems are down. Given Github's scale and number of engineers it's basically guaranteed that they'll hit some sort of bug like that eventually. Not all bugs act immediately.
Re:Improving how we deploy GitHub by notwhereyouare on Monday January 25, 2021 @04:22PM (#25909086)
I did a short stint at wayfair and about 1-2 months in, there was a deploy that somehow got passed the test flow and when deployed took down their entire site. So badly that they couldn't even deploy the fix
Re:Improving how we deploy GitHub by cutemonster on Tuesday January 26, 2021 @08:37PM (#25923238)
How did they solve that in the end?
Re:Improving how we deploy GitHub by Existenceblinks on Tuesday January 26, 2021 @11:57AM (#25918159)
It sounds like a good idea, but is very bad one. I worked on a git deploy product and dogfood was very appealing, turned out it's confusing as fuck. And When things get wrong it's even more confusing, it's like horror time travel movie when you can't find origin to fix consequences.