Re:Stack Overflow: How we upgrade a live data center
by aalear on Thursday March 05, 2015 @11:18AM
(#9151639)
Basically, yes. This happened: https://twitter.com/atimmer10/status/560568418945236995.
Re:Stack Overflow: How we upgrade a live data center
by xtrumanx on Thursday March 05, 2015 @01:03PM
(#9152553)
How long did it take to figure out it occurred? Do you have any stats on how many people hit that machine?
Would be hilarious if someone who went to SO to figure out why the IIS splash was showing instead of his app happened to hit that page.
Re:Stack Overflow: How we upgrade a live data center
by GeorgeBeech on Thursday March 05, 2015 @01:17PM
(#9152677)
It took about 20 minutes to notice and fix. The Twitter alerting system let us know really quick.
Re:Stack Overflow: How we upgrade a live data center
by bagels on Thursday March 05, 2015 @03:05PM
(#9153450)
Why is their load balancer not configured to take servers like that out of rotation?
Re:Stack Overflow: How we upgrade a live data center
by Nick-Craver on Thursday March 05, 2015 @03:33PM
(#9153621)
It normally would, except the base IIS page is a proper 200 to all requests. The load balancer removes all servers having issues - from HAProxy's point of view things were dandy.
As always, we are looking how to adjust our deployment process to prevent it from happening in the future.
Re:Stack Overflow: How we upgrade a live data center
by clogston on Thursday March 05, 2015 @05:02PM
(#9154155)
Everyone I know big enough to run a load balancer has had this happen.
The next step in evolution is to have your script look for specific text on a page, which will change 24 months later and have all the perfectly good servers pulled out of rotation.
Re:Stack Overflow: How we upgrade a live data center
by xxpor on Friday March 06, 2015 @12:14PM
(#9158151)
Does bare IIS respond with a 200 to /ping or /sping?
Re:Stack Overflow: How we upgrade a live data center
by GeorgeBeech on Thursday March 05, 2015 @11:20AM
(#9151651)
We generally pick a hashtag on twitter and do it that way. We've also done Hangouts On Air for non-hardware in the past which have been pretty fun as well.
Re:Stack Overflow: How we upgrade a live data center
by gbrayut on Thursday March 05, 2015 @12:15PM
(#9152136)
If by fun you mean a 4 hour maintenance window turning into 8 hours ending well after midnight :-P
Re:Stack Overflow: How we upgrade a live data center
by GeorgeBeech on Thursday March 05, 2015 @12:47PM
(#9152413)
I never claimed to be sane
Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO?
I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.