Re:Facebook-owned sites were down
by jaywalk on Monday October 04, 2021 @04:15PM
(#28751507)
Why? Even if it's not DNS reliance, if they self-hosted the server (very likely) then it'll be just as unreachable as everything else within their network at the moment.
[ Parent
| on HN ]
The entire purpose of an IRC backup is in case shit hits the fan. That means having it run on a completely separate stack.
What use is it if it runs on the same stack as what you might be trying to fix?
Clearly "our entire network is down, worldwide" wasn't part of their planning. Don't get too cocky with your 20/20 hindsight.
I don't think it's cocky or 20/20 hindsight. Companies I've worked for specifically set up IRC in part because "our entire network is down, worldwide" can happen and you need a way to communicate.
I bet they never tested taking out their own DNS.
IRC does use DNS at least to get hostnames during connection. I'd be surprised if it didn't use it at other points.
I’ve setup hosts files in case DNS was down to access critical systems before. It’s a perfectly reasonable precaution.
My small org, maybe 50 ips/hosts we care about, maintain a hosts file stills, for those nodes public and internal names. It's in Git, spread around and we also have our fingers crossed.
Re:Facebook-owned sites were down
by littlecranky67 on Monday October 04, 2021 @04:41PM
(#28751813)
If only IRC would have been built with multi-server setups in mind, that forward messages between servers, and continues to work if a single - or even a set - of servers would go down, just resulting in a netsplit...Oh wait, it was!
My bet is, FB will reach out to others in FAMANG, and an interest group will form maintaining such an emergency infrastructure comm network. Basically a network for network engineers. Because media (and shareholders) will soon ask Microsoft and Google what their plans for such situations are. I'm very glad FB is not in the cloud business...
> If only IRC would have been built with multi-server setups in mind, that forward messages between servers, and continues to work if a single - or even a set - of servers would go down, just resulting in a netsplit...Oh wait, it was!
yeah if only Facebook's production engineering team had hired a team of full time IRCops for their emergency fallback network...
Re:Facebook-owned sites were down
by littlecranky67 on Monday October 04, 2021 @05:53PM
(#28752562)
Considering how much IRCops were paid back in the day (mostly zero as they were volunteers) and what a single senior engineer at FB makes, I'm sure you will find 3-4 people spread amongst the world willing to share this 250k+ salary amongst them.