Re:Facebook Down?
by erik_seaberg on Thursday April 08, 2021 @06:35PM
(#26744652)
5 is often broken down to “rain fade” and “backhoe fade.”
My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”
I was going to link a related comment but you're actually the author of it!
Here it is anyway: https://news.ycombinator.com/item?id=22316230
Re:Facebook Down?
by reaperducer on Thursday April 08, 2021 @06:43PM
(#26744722)
Coincidentally, I'm in the same predicament with Facebook, itself.
I can't log in to Facebook because Facebook wants approval from the Duo Mobile 2FA app. The 2FA app won't approve the login because it wants a code that can only be obtained by logging in to Facebook.
Solution: Deleted both Duo Mobile and Facebook apps. Moved on with my life.
isn't that like throwin the baby out with the bath water? i've never had problems with duo mobile, but then i don't use FB so i've never seen how it behaves with FB. so dump FB, but why dump duo mobile?
Re:Facebook Down?
by reaperducer on Friday April 09, 2021 @11:58AM
(#26752215)
Because the only thing I used it for was to login to Facebook. All of my other 2FA is through other methods.
Re:Facebook Down?
by lovedswain on Thursday April 08, 2021 @06:36PM
(#26744671)
In a similar vein, the alternative option for 7 was a cyclical dependency introduced since last restart
Re:Facebook Down?
by erik_seaberg on Thursday April 08, 2021 @06:46PM
(#26744754)
I heard a story that Google SRE had to up a sacrificial Bigtable in any new zone before they could up enough services to finally up a real Bigtable for the zone.
Please, please tell me this is a true story. It's amazing.
Re:Facebook Down?
by erik_seaberg on Thursday April 08, 2021 @08:39PM
(#26745522)
I can only verify being unable to get the wi-fi password. Otherwise it’s all secondhand, I wasn’t on call.
Re:Facebook Down?
by ASalazarMX on Thursday April 08, 2021 @05:58PM
(#26744347)
> Wolves ate through fiber - 5 shots
I hope it's wolves, that would create a new category in https://cybersquirrel1.com/
Re:Facebook Down?
by partcyborg on Thursday April 08, 2021 @06:09PM
(#26744446)
You forgot:
Hunter shot your fiber relay with a hunting rifle - 6 shots
6 shots? It only needed one.
depends on the gun... some people have interesting ideas on hunting.
Re:Facebook Down?
by godelmachine on Thursday April 08, 2021 @06:14PM
(#26744486)
DST can be such a hassle.
It affects our reports and change requests.
Re:Facebook Down?
by erik_seaberg on Thursday April 08, 2021 @06:37PM
(#26744676)
An adtech company once freaked out over a sudden 4% revenue drop, until someone pointed out that yesterday was 4% shorter than the day before.
Re:Facebook Down?
by alfalfasprout on Thursday April 08, 2021 @06:36PM
(#26744665)
Don't forget cosmic rays!
bad DNS config push - 1 shot
routing loop - 2 shots
third party advertising your routes - 3 shots
power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots
Wolves ate through fiber - 5 shots
And it was a full moon - 6 shots
Single service failure, but service has not been restarted in 5 years, and no longer restarts in any documented fashion - 7 shots
And service developers left the company to found a startup - 8 shots
Expired internal SSL certificate - 9 shots
Daylight savings changeover-induced database corruption - 10 shots
Windows Update - 11 shots