In many ways, it's beautiful in that it allows Autonomous Systems to come together and independently make the Internet.

However, I always felt it was brittle in that it relied every AS to have edge routers aware of the entire AS routing map, and so when those routers went down hard, when they came back up they were blank and needed to learn the Internet again. I noted back then that on some routers a single spoofed UDP packet to a router getting a table from its peers could cause it to stop doing that and start again. In a time when those updates could take hours... well... that kept me awake at night sometimes.

Propagation of routes was global and error prone. Even today, you can see on the spy glass websites rogue AS routers advertising crazy routes either by accident or on purpose as a hostile act (to try and get traffic for a target network through itself).

There are tens of thousands of engineers globally nursing and managing this stuff to keep the whole thing going.

Like I said, it's been a while since I last looked closely, and I imagine multiple improvements have been made in this space since that time, but if there was ever a protocol that needs a long, cold hard look for a replacement, it's quite possibly BGP.

It's also the one - and only - non-crypto situation where if somebody asked me if a blockchain could be really very useful, I'd say "probably, yeah".

Human readable to IP: DNS

Private IP to Public IP: NAT

MAC to Private IP: ARP

Public IP to AS: BGP

With a block chain DNS it may be possible to replace all of them with a single protocol, but it may well turn out to be an academic exercise.

That deals with DNS (though how much has it been used?), but what about the layers of the stack?

[1] https://en.wikipedia.org/wiki/Namecoin

[2] https://bitcointalk.org/index.php?topic=1790.222

Most of computing is about mapping A to B and maintaining databases of those mappings :-)

Most of these systems are about naming things and cache invalidation.

I could maybe see a use case for DNS but then either each computer has to have the full blockchain locally or rely on providers to do that for them in which case we're basically at the same model we have now (with root servers and DNS servers).

Obviously you don't want to publish private IPs, etc. to other computers... so inside a local network you're not buying yourself much.

For routing tables I could see more of a use case for blockchain technology since you can actually verify the routing tables and prevent BGP spoofing, etc.

For anyone that's interested in what these may be. Take a peak at MANRS. https://www.manrs.org/

Reconnect and the GLBs fall apart under load as the entire world’s cadre of recursive resolvers hit you.

Fix that. Reconnect again. This time your LBs have marked half the servers as offline because their heartbeats have been failing.

Fix that. Reconnect again. Now all the memcache data is hours old and so the site business logic fetches straight from databases, knocking them over.

Fix the databases. Reconnect again. Ad nauseam.

I can’t imagine trying to restart something as big as facebook…

Facebooks no longer an innovator, just a mining operation with a dwindling population of hateful elderly and bots.

1: https://image.slidesharecdn.com/datareportal20210308gd001dig...

2: https://image.slidesharecdn.com/datareportal20210308gd001dig...

3: https://wearesocial.com/blog/2021/01/digital-2021-the-latest...

Having been a part of Google the only thing more awe inspiring than the sheer complexity of production is the fact that it all worked so well.

This is not a dig at current Googlers but entropy is cruel and uncaring. Perhaps parts of the stack which have been kept fit & fresh in people's minds due to constant rewrites will last longer but there are tons of places in the depot that are unowned despite serving production query traffic and the number of engineers that have any context to support it grows smaller over time.

No idea if Facebook is similar.

It's pretty inexcusable that FB wasn't able to use OOB management.

I suppose with something like BGP it would be very difficult to get such a fallback working given how distributed the system is, and even more difficult to keep it exercised and tested.

https://www.juniper.net/documentation/us/en/software/junos/c...

But Facebook's network is certainly much more complex and automated than just doing one commit on one device.

But I think they do still uses Juniper devices at the edge.

<snark> Software defined networking. In PHP.

iptables-apply reverts to previous config instead of flushing all the rules.

How do you know if you waited long enough to see negative effects (think caches, your own and caches of others).

Waiting too long with a bad config also costs you.

Whereas to make an announcement, the entire internet (or at least all routers between the AS and the user) need to pickup the new announcement.

(Note: I still need to read the article)

I think it's not so simple because authoritative DNS systems are involved.

So it's not just a BGP error. It's a BGP error which disconnected authoritative DNS for all facebook. I'm not quite sure why that makes it so slow to fix. is it just because internal difficulties due to having no DNS at all?

I would find it a bit surprising if Facebook didn't have OOB access to their data centers, however.

Next time FB save you passwords in OneDrive and Google Drive as a backup LOL. [email protected]

Laptop + mobile tethering + serial cable to the router + teamvewier for the remote admin to get the access solves problems like this in minutes.

Breaking a gajillion security policies by doing that is a different story though.

It's an IP address management (IPAM) solution that also just happens to be a fantastic, federated (if you want) DNS management system too. Indeed a previous org I worked at bought it strictly to tame the DNS beast - local sys admins could control DNS for their subnets but not affect anything else. If we wanted, we could have had approval processes on top of the change requests - the system supported that too.

I think the security teams finally woke up to the IP address management functionality and were slowly starting to integrate that into the rest of the infrastructure - but I was leaving around then. It was a fantastic system. One of the best hierarchical role-based access control systems in an application I have ever seen; the granularity was amazing yet it was easy to understand/administer. Not an easy trick!

I don’t know enough about BGP to make an informed decision; but at the point the outage is noticed it’s entirely possible that the system has been unavailable for quite some time already.

https://ns1.com/resources/dns-propagation#:~:text=DNS%20prop....

(There's a corner case related to DNSSEC that can make it go higher, but that's being worked on, and isn't relevant here.)

In this situation, the nameservers were just down. I haven't done exhaustive research, but the resolvers I'm aware of cache that kind of thing for no more than 15 minutes.

You are feeling hungry and went to food court. The food court (open area) has a lot of options. You sit down in front of Domino's (Facebook), since you want to eat garlic bread. Now, you can't order from the counter directly. The waiter will come to your seat and ask for the order. You ordered garlic bread from the waiter, but the guy at Domino's counter went missing. Your order was not reaching to the chef in kitchen as Domino's counter guy was not present.

This explains why Domino's (Facebook) ecosystem was down, but what about other vendors? They had nothing to do with Facebook.

To understand this, we need to go back to our food court again. Now, there are a lot of hungry people sitting outside Domino. Since they were not getting answer from one waiter as why their food is not on their table, they started disturbing all the waiters. Due to this, majority of the waiters were trying to figure out where the Domino's counter guy went and other food joints (read websites) were not able to fulfil their own orders.

So although only Domino's was down, it appeared as if whole Food Court (Internet) was facing issues.

Counter Guy at Domino's - Facebook Nameservers Waiters - DNS Servers (Cloudflare, Google, Akamai)

Facebook being down was already an issue, but everyone phoning and knocking on doors was causing disruption to everyone else.

As result, the borg collectives can access to network, even 7of9 can't enter the campus.

Internet is just a bunch of computers interconnected via tons of cables (hence the name; "inter-networked computers").

To be reachable, every equipment and computer constantly need to tell the others about their existence (to publicly announce on which network cable they can be reached at).

Facebook engineers wanted to optimise that system but accidentally broke it during the update.

As a consequence, after a few minutes, other computers didn't know on which cables they can reach Facebook.

Facebook had to call the technicians sitting in the datacenter to cancel the last change that was done (because the Facebook engineers couldn't themselves connect from the office) and everything was fine again.

Is that right ?

That analogy includes the snowball impact on the other websites and services as the Switchboard Operators get more over-utilized into puzzling out Facebook's problem than servicing calls for still working phone numbers.

Mum, dad - you know how I always tell you to turn your stuff off and on again?

Well, by turning it off, Facebook also turned off the On Button.

Someone assumed London==UTC, when London is 1 hour ahead :) that was actually 21:28 UTC

It's especially annoying when dealing with multiple countries that may or may not be using Daylight Saving Time.

A few weeks ago I tried to find out what the current time in CET is. Asking google for "CET" gave me: "23:27 CET". Asking google for "CET time" (I know that "time" is twice in this case) gave me "00:27 CET".

The last one is wrong and should be CEST or even more correct would be just the same result for CET as I asked for

    $ zdump -v Europe/Berlin | grep 2021
    Europe/Berlin  Sun Mar 28 00:59:59 2021 UT = Sun Mar 28 01:59:59 2021 CET isdst=0 gmtoff=3600
    Europe/Berlin  Sun Mar 28 01:00:00 2021 UT = Sun Mar 28 03:00:00 2021 CEST isdst=1 gmtoff=7200
    Europe/Berlin  Sun Oct 31 00:59:59 2021 UT = Sun Oct 31 02:59:59 2021 CEST isdst=1 gmtoff=7200
    Europe/Berlin  Sun Oct 31 01:00:00 2021 UT = Sun Oct 31 02:00:00 2021 CET isdst=0 gmtoff=3600

(And that's not counting daylight savings time, and its varied observance!)

No joke. Today I ended up writing a whole essay explaining the issue I was having and almost sending it off to the core developers because I thought I had discovered an issue with the actual language. The bug was because I had forgot to convert too&from utf-8 in these two procedures:

    proc 2Hex { input } { binary encode hex [encoding convertto utf-8 "$input"] }
    ;# Converts base32 string data to base16

    proc 2Base { input } { encoding convertfrom utf-8 [binary decode hex "$input"] } 
    ;# Converts string hex data to base32

The most "natural" occurrence that I can think of is best-path change. If a "better" route between AS is added, the now-second-best routes are withdrawn.

Correct me if I am wrong, but there is no way of determining the source of the withdraw message (UPDATE)...

and

> "... its root cause was a faulty configuration change on our end."

From https://engineering.fb.com/2021/10/04/networking-traffic/out...

No key-card access means even non-need-to-know internal employees wouldn't see the deed, and plausible deniability is spawned for everyone else.

Edit: Also a great way to suppress the news of the whistleblower. Take the site down so the outage outranks news searches keyword "Facebook"

What's scary is that my gaming machine can't be used because of some weird obsession with centralizing oculus through facebook. As a customer that's inexcusable but now imagine a developer who hosts her own servers for her game and her customers can't get to it. Facebook is only providing middle-man authentication, its not even hosting these games. If they were hosted by FB it would be fine, but they aren't and what we're suffering under is FB being the gatekeeper to the rest of the internet. That is a scary prospect.

I also question why we all think its acceptable to have these incredible BGP outages every 3-6 months. We built our digital world on the equivalent of ever changing spinning plates and manage it in the most cost-efficient way possible. Maybe the alternative would been worse, but its crazy to me as to what we consider normal and acceptable in capitalist culture.

I'm already seeing /r/oculus say its no different than Valve's scheduled maintenance windows, which is obviously false. Its incredible what rationalizations we'll accept instead of questioning the status quo of capitalist culture and our giant corporations that rule so much of our lives.

This isn’t BGP’s fault, someone or something presumably made a network configuration change that took down their route advertisements to the world. There’s currently ~72K autonomous systems advertising ~900K IPv4 prefixes today on the Internet. There’s bound to be some sort of screw up once in a while.

AR Glasses at least their form factor has changed since Google Glass and will continue to evolve some. Google Glasses to how Facebook Stories sunglasses look which look like sunglasses billions are use to wearing and use daily. Billions will never strap on a headset for hours to co-work. Billions will adopt AR Glasses as it takes a familiar form factor/daily life product and enhances it like the iPHone enhance our daily lives.

But, seriously, it could use a separated network with federation like an AD domain.

And if they don't mitigate it now, every hacker with access into their network now knows how to bring the whole show down in an instant.

The instagram.com zone itself uses a third-party DNS service and didn't go down. (But e.g. www.instagram.com is a CNAME to a zone on FB DNS.)

The article itself was a good exploration of the impact of BGP and what happens when network advertisements stop and the associated network disappears in a puff of global forgetfulness.

Might be time to poke around BGP as well. A lab setup might be a good toy.

The idea is you do something, you messed up and are locked out, the system will revert to previous config after a few minutes by itself. The story of people physically accessing systems only makes sense, if this was a hack...

===============

Commit Confirmed

"Suppose that despite all your efforts to insure your new configuration is correct before you commit it, something is overlooked and when you commit, you are locked out of the router."

"Rather than just a simple commit, you can make the candidate configuration active with a commit confirmed command. With this command, the router waits 10 minutes for a second commit. If it does not receive that confirming command within those 10 minutes, the router automatically does a rollback and commit so that the previous configuration becomes active again."

https://www.networkworld.com/article/2345600/managing-a-juno...

"Committing a Configuration"

https://www.juniper.net/documentation/us/en/software/junos/c...

===============

Some other internal issues could have broken many systems, including BGP.

They could have had some sort of internal systems failure, and intentionally withdrew BGB to cut off the flood of connection attempts (making recovery easier).

Many possibilities.

Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing glorified bikeshedding, and providing an explainer of how BGP works that can be gleaned by reading virtually any introductory text on it, and using it as a marketing opportunity.

I mean, I'm not really sure what the purpose of a corporate blog is except that? You make posts about whatever will garner attention in order to get some views, maybe spread some information, and - of course - turn it into a marketing opportunity. That's the job, no?

> Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation?

Personally? No. Although amusingly, a non-technical friend of mine took the twitter posts they made as a sign that Cloudflare had caused the outage somehow, so it's certainly possible there's a risk there.

And from one of your other comments:

> Next time Cloudflare's CDN eats itself and starts vomiting up private customer data, Facebook can do a blogpost titled 'Understanding how Cloudflare exposed the private information of untold numbers of its customers'.

Yes, they should totally do that (at least if they have anything informative to contribute, as I think Cloudflare does here). Why would this be a bad thing, or a reason not to talk about Facebook's issues? And I mean, at one point today something like 6+ of the top 10 links on HN were about Facebook, so I mean, everyone else is talking about them. Why not Cloudflare? And if and when Cloudflare has their next big issue, everyone will be talking about them either way.

> We keep track of all the BGP updates and announcements we see in our global network. At our scale, the data we collect gives us a view of how the Internet is connected and where the traffic is meant to flow from and to everywhere on the planet.

> Fortunately, 1.1.1.1 was built to be Free, Private, Fast (as the independent DNS monitor DNSPerf can attest), and scalable, and we were able to keep servicing our users with minimal impact.

It isn’t a big deal, and the posts are still interesting. It just makes me roll my eyes a bit.

I very much prefer that over the almost patronizing, overly friendly tone some others have, or the stripped of any personality style that most have.

You kinda inadvertently highlighted the issue: because they had nothing to do with it, they do not know what actually happened. They can pontificate about likely causes, just like others in the industry can, but they have no idea what actually caused the issue.

You can pontificate about likely contents of Cloudfare’s blog post, just like others who did not read it, but clearly you have no idea what it actually contains

> They can pontificate about likely causes

They can, but they didn't.

Generally big companies don't talk smack about other big companies having internet-wide issues, unless those issues are directly caused by the other company.

For instance, when Google talked about Cloudbleed, which was when Cloudflare vomited millions of secrets all over Google's caching heirarchy and Google had to manually clean it up.

I think perhaps the Cloudflare people have gotten confused and think that means it's okay to talk about other people's stuff. Instead of interpreting it as it really is, which is that Cloudflare is the last company that should be criticizing everyone else, lest someone bring up their previous missteps.

There is no criticism of Facebook in our blog post.

Though I would expect something like this from Fastly more than Facebook.

At the fact that they are presenting themselves as authorities on the incident, when they have no internal knowledge of what triggered the events, because they are not Facebook engineers. They provide an explanation of BGP that you can glean by reading virtually any other introductory explainer, and turn it into a chance to promote their own service.

This is not an honest representation of the article.

They talk about what their services observed related to bgp traffic from facebook. They are an authority on that.

They talk about their dns traffic changes from facebook's outage. They are an authority on that.

They talk about suspected causes, based on the observable data, and guess what, given they are who they are, this is something they are a subject matter expert on.

"we saw a spike in bgp traffic from facebook followed by a bunch of route withdrawals. we think this could be a bgp configuration issue [given we took large chunks of the internet down 2 years via the same fuckup]"

Is something in the subject matter wheelhouse of cloudflare, yes.

Sample size of 1, but I imagine that's more or less the reaction they're hoping for.

They talk about their dns traffic changes from facebook's outage. They are an authority on that.

With the new DoH Chrome setting turned on by default (DNS-over-HTTPS), AFAIK they are now the default resolver bypassing your network settings.

[1] https://developers.cloudflare.com/1.1.1.1/encrypted-dns/dns-...

[2] https://duo.com/decipher/google-makes-dns-over-https-default...

Nothing is "explained" other than what we knew already, that some unlucky SOB shot themselves in the head with a BGP shotgun.

I understand, they have no way of knowing what happened inside Facebook. But they could give some detail about exactly what the BGP updates were, the structure of the IP space served etc.

That's kind of incredibly disrespectful to even compare the two

https://krebsonsecurity.com/2021/10/what-happened-to-faceboo...

If you downgrade from a Paid to a Free account they automatically and silently add a paid add-on for 10 Page Rules.

> If you do not want to be charged for the additional page rules, you should ensure you only have 3 active rules before you downgrade from Pro to Free.

[1] https://community.cloudflare.com/t/i-canceled-my-subscriptio...

And pretty sure that explanation is incorrect or has changed. I have never used Page Rules and was still billed.

Fair point.

Do you want to update BGP?

No: exit

Yes: type this random 100 character phrase to continue no copy paste

I've seen mentions everywhere that to fix this they required physical access to the BGP routers.

My understanding is that if a bad BGP route was pushed and your whole network (typically accessed via VPN while the engineers WFH) is probably unreachable from any of you’re employees’ homes, it’s hard to get the right people on site to make a fix, or talk the remote hands folks at the date center through it over the phone. Troubleshooting is hard, troubleshooting while you have no access to your own network has got to be just that much harder.

The fact that DNS was also not resolving was a symptom of the DNS servers also being unavailable since they were part of the same IP address space that was unadvertised.

When it comes to Facebooks side I guess they do have backups of their BGP config. Applying them (probably remotely) however seems to be harder then expected when the whole infrastructure is down.

they basically ran into a footgun

It's good that we have some coverage from companies that have some stake in the game because they are also affected by the outage, even if only partially.

I expect the detail level to be roughly "an automated system pushed a broken configuration"; that is to say, there probably won't be any interesting information at all for the Hacker News crowd.

I doubt that this was caused by "hackers" or "hostile governments" or "dissident employees upset about Facebook privacy issues", and also doubt that Facebook would admit such if it were true unless they were legally required to do so.

History has shown us they can give us zero response, or an incorrect response, and we (via our representatives) will accept it and continue living life as before.

The guys with the blue windbreakers show up, I'd pretty much say "yes, sir." Of course, I don't have FB's power, but I don't think it matters.

shareholders aint much these days of cancerous growth

Yet another reason to dismantle it.

Not everyone uses Facebook services and the rest of the internet worked just fine.

https://www.theguardian.com/technology/2021/sep/29/facebook-...

Did Facebook lie to congress?

https://www.forbes.com/sites/zakdoffman/2019/03/02/facebook-...

Also short comments without discussion and evidence do poorly.

I believe it should be halted for public safety which is well within the rights and capacity of our government. I believe a fair trial should happen before anyone receives punishment, but I don't believe we could find an impartial jury as Facebook is ubiquitous.