I work in one of these companies and I can confirm that is a network outage and even our internal tools hosted in our datacenter (so no dependencies with fb) are down

I sincerely hope that AS3356 hasn't broken a big chunk of the Internet again.

https://isc.sans.edu/forums/diary/CenturyLink+Outage+Causing...

https://www.zdnet.com/article/centurylink-outage-led-to-a-3-...

My bet is BGP. Its always BGP.

It's been a while since I studied routing, but have they fixed BGP security?

I remember reading about the protocol and thinking something like: this seems awfully "hippie-trust" levels of security.

Granted, this was 10 years ago so my recollection of BGP is super fuzzy :-D

nope. proper and full implementation of BCP38, MANRS, full RPKI validation of all IP ranges and such is a long distance away.

within trusted groups of ISPs where the admins all know each other, in certain specific geographic regions, it's better than others.. I would say that the number of ISPs I could call "fully compliant" with current best practices, in the Portland-Seattle-Vancouver area, is higher than in many other parts of the world.

How is shaw?

Anecdote: I was surprised to find they honour TTLs on their DNS service. Everyone else I have access to seems to return the new IP almost immediately, but shaw waits it out.

As someone interested in the internal workings of the internet, do you have any further reading on current best practices?

ARIN, RIPE and APNIC publish a great deal of useful documentation on how to set up BGP stuff properly in the modern era. And information for how to fully implement dual stack with ipv6.

As for other current best practices there's really a very great variety of disciplines involved in ISP operations, depending on what type of ISP it is. What's relevant for a GPON FTTH last mile provider might not be as relevant for a colocation/dedicated server ISP. Or for a WISP. My recommendation on that would be to pick the subject to research and make inquiries in the special interest groups dedicated to that (for instance, two way satellite, or automated billing/provisioning systems for hosting companies, or CRM systems, or whatever).

To be fair, BGP only runs between large network operators. The security / trust of routing agreements is mostly solved through business deals and contracts.

Every time this happens I have a mental metaphorical image of somebody tripping over the $5 power strip that has the single point of failure route-server or route-reflector plugged into it.

I stand corrected :)

Any info on what network/operator/operator/etc is suffering the issue? (just so people know about it if it's public, i don't see anything on HN at least)

Thanks, and good luck!