Or they felt like some of us feel about Tesla/Musk unleashing cars with "Full" "Self-Driving" on public roads.
There are other people building driverless cars, who have safer systems that they rolled out reliably.
Musk isn't saving people from drunk drivers, he's trying to catch up with the competition and chose to kill people to catch up.
We’ve been building trillions of dollars of public transit since about the same time as we started building the National highway system. So when exactly does public transit start saving all these lives?
A new train from SF to LA has been in the works since 1996. They are expecting partial completion for $30 billion by 2030 which is projected to carry ~15 million passengers * ~400 miles = 6 billion passenger miles per year.
The fatality rate for passenger cars is ~0.57 per 100 million miles, so eliminating 6 billion passenger miles could be reasonably expected to save about 60 * 0.57 = 34 lives a year out of the ~40,000 that are dying on the roads annually.
This is just rough napkin math showing the scale of the problem would require a roughly $30 trillion investment in public transit to appreciably shift passenger-miles away from passenger cars.
Public transit can be very useful. But it’s clearly not the solution to eliminating driving deaths.
You argue that it would be very expensive to fix the problems we've made for ourselves in order to get useful public transportation and that's certainly true. It's also very expensive to invent self-driving cars and part that cost has already been paid with lives lost and human suffering, but that hasn't stopped us from developing those systems. We could decide to invest instead in improving our cities and infrastructure. We just choose to let obscenely rich people kill innocent Americans while they play with their profit generating spying car tech instead.
I suspect a Network Operators mailing list is going to be full of people trying to keep production up, with the tools they've got, rather than the tools they'd like. Thinking the software quality on the routers they're using are dogshit is probably just called "Monday" and thinking their peers' routers are even worse is called "Tuesday".
I can understand they don't appreciate "research" done on production systems that might cause issues when they already know their equipment sucks.
Bitching at researchers (note: not "researchers") who are finding and calling out those flaws before someone really bad comes along and exploits them is somewhere between unhelpful and negligent. Internet security is always improved by more sunlight.
I think you fail to realize that the internet is a bunch of disparate networks that no one person controls. When "fixing" a router means you break half the internet, you just don't fix it. This isn't some walled garden where you can just say "tough shit if you broke, fix it".
Most of the "issues" these researchers find are well known, but backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.
The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed. Once a vulnerability is known then someone out there is going to start exploiting it almost immediately.
> backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.
Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP. This is a security lesson that has been learnt in recent years by many sectors of the IT industry and, judging by the woeful response that Ben Cartwright-Cox describes in his post, it's one that the network backbone is going to be learning soon.
So you've never worked in a corporate environment. Here's how that conversation would go:
Hey guys, some researchers found out if you use a test flag meant for the lab on the public internet, it breaks all our BGP sessions.
OK, so drop their feed and block them?
Great, done.
>Once a vulnerability is known then someone out there is going to start exploiting it almost immediately.
And yet the "vulnerability" in question was known, and was not being immediately exploited if you read through the mailing list or were participating in NANOG at that point in time. So your statement is provably false.
>Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP.
And yet we're having this conversation in 2023, they have operated the same way for 40+ years, and somehow the internet is still working. Bad actors get blackholed, it worked in the past, it'll continue working in the future. The reality is that backbone routing is expensive, and expecting everyone to update their kit on YOUR timeline isn't reasonable.
I am prophesying that a similar reckoning is likely to come upon the network backbone. You're arguing that the cost of entry to the game (BGP peering) is such that the old ways will continue to work. Let's hope you're right.
Practically, I accept that there is a real difference. ASNs are much harder to acquire than IP addresses, and there are far fewer of them. That difference might mean that blocking continues to be an effective mechanism. However, in a world where malicious actors are increasingly just nation states wearing a flimsy mask, I'm not sure that the difficulty of getting access to sufficient numbers of ASNs is going to be a forever mitigation.
The main thing I have taken away from this thread is that when it comes to vulnerability management, network operators seem to be pretty far behind the curve in mindset terms. The arguments being made in here ("you can't break other people's stuff", "we have a lot of boxes and SLAs mean we can't patch quickly") aren't even slightly new. The well-proven reality is that threat actors don't give one single shit about any of that.
> Great, done.
The "great" is the problem, no?
Bearing in mind we also couldn't just "apt-get upgrade" in place, most boxes required hard reboots to apply the patches.
The answer to your question is no, as per a few others have said, Don't violate Rule #1. We see bad actors very often, our job is to keep the bits flowing and the internet online.
Keeping the internet online is painful enough as it is without "researchers" dropping thousands of routers to "prove a point."
Research is great but if you're poking around breaking things via DoS in the internet you're going to be judged as a bad peer regardless what the route daemon does. The security of the internet constantly moves forward but that doesn't mean the expectation of improving has to hold more precedence than the expectation of running. It's still very much a human to human system for all but the actual route exchange.
We call out vendors when their quality suffers but you only have two or three practical vendor options for high end routers depending on your use case/feature requirements. All three have varying cycles of hardware and software quality, with hardware and software rarely being in phase for the same vendor.
The fact that all three vendors make most of their money on non-network focused enterprises with less sophisticated engineers doesn't help get development attention where it needs to be for global infrastructure.
Overall I agree with your message though, and of course things shouldn't stop working just because you use attributes reserved for development.
[0] https://mailman.nanog.org/pipermail/nanog/2019-January/09876...
Thou shalt not blow up another's network knowingly.
Blowing up another's network because you didn't test your theory in your own isolated lab space first counts as blowing up another's network knowingly. Because what the hell else did you think would happen?
Cripes.
Dropping the whole session, especially in a DFZ scenario is really bad. The churn that you generate is immense for all your neighbors.
BGP is one of the most robust protocols out there: there's no state (as in protocol state), you can treat each message independently, you can extend it super easy and so on.
And because you can treat every message independently, you can have a really secure and stable mechanism to deal with any badness that might happen.
GREASE means that when you build your "advanced" security device which "protects" customers by treating everything different as hostile, it won't work, so you'll need to tweak it to at least just ignore such differences as irrelevant, which is enough that we can come back later and intentionally improve things.
Previously, without GREASE, we'd have to guess what oversights we could exploit to avoid this "protection" to deliver protocol improvements, if we guessed wrong nothing works, or everybody's security is broken, sometimes both. e.g. for TLS 1.2 the oversight we found is, if you're resuming an existing session the "security" products just wave that through because otherwise they break people's real workflows. So in TLS 1.3 protocol as actually spoken essentially an initial connection goes like this:
Client: Hi some.dns.name.example I'm a TLS 1.2 Client, I'd like to resume our previous conversation #randomNonsense. Also, completely unrelated, I happen to speak FlyCasualThisIsReallyTLSv1.3 and so I have these TLSv1.3KeyAgreementParameters.
then either:
TLS 1.3 Server: Hi Client, of course, let's continue from there. [ whereupon everything further is encrypted because this is actually TLS 1.3, but to a dumb middlebox it makes sense that they're just resuming a prior encrypted conversation #randomNonsense which it doesn't remember ]
OR
TLS 1.2 Server: Er, no I don't remember any such conversation and I don't know FlyCasualThisIsReallyTLSv1.3 so let's start a fresh conversation as normal.
If you are, good on ya. Continue doing $deity's/Bob's work.
If your system is on the internet it is now a potential piece in a potential global war.
We've blasted past that age at this point, do everybody gets to don their robe and wizard hat, pick up the staff, and yeet patches, routing rules, and packets betwixt one another.
1) Beleaguered people misbehave.
2) You are usually the architect of your own destruction.
3) It’s easier to see other people’s problems than your own.
Stir into a death spiral and let it rest for six to twelve months. Bake at 375° for two years, then serve cold with a sprinkling of schadenfreude to offset the bitterness.
Does any sane person think that 1 week is enough time to both notify every user of FRR in the WORLD, as well as ensure they have upgraded their installations for a "bug" that only is triggered by someone using experimental flags meant for testing purposes only?
Mind you, these researchers didn't test ANYTHING in a lab environment first "because it's a lot of work to test all the open source BGP implementations in the wild". AKA: I'm lazy.
That's a gross exaggeration: they said that they first tried it "in a controlled environment with different versions of Cisco, BIRD, and Quagga routers" [0] without issue. Presumably, the problem we know about would have been avoided if they had happened to include FRR in their test suite beforehand. But that raises the next question, exactly how many implementations does one need to test before one is no longer "lazy"?
[0] https://mailman.nanog.org/pipermail/nanog/2019-January/09876...
The outrage here is understandable. I would be pissed if I was paged because halfway across the world, some asshole caused this intentionally. Also this mailing list is aimed at the operators, not the vendors. Different communities.
The vendor response here, I do agree is irresponsible. Besides OpenBSD, every one here just sucked.
- I think it's OK not to want experiments like these to be conducted connected to the global internet and rather in a lab such as the one benjojo used.
- NANOG is known to be a bit of a joke within the networking space, not too long ago someone complained about NTP issues with traffic passing through TATA and they were told to make their devices stratum 0 or something to that effect, so don't always take them too seriously.
I think the first applies here.
Not saying your summary of that thread is wrong, I just wanted to expand a bit to maybe clarify:
- someone posted to NANOG about issues with their (European) servers reaching 0.freebsd.pool.ntp.org
- they noticed they were being routed through Tata (AS6453)
- they mentioned that their other servers (e.g. in Africa) had no issues
- they posted to NANOG to ask about this
- they thought it was a routing or FreeBSD issue (they later posted about the issue to a FreeBSD list as well)
So AFAICT, the correct response was posted about a day and a half (~42 hours) later: - the NTP pool uses GeoDNS to map clients to servers
- the OP's server IP address was geolocated to Mauritius
- the Mauritius zone in the pool has only one server
- so the OP would only ever get that IP as an answer
The short-term fix was to use a different pool address (e.g. europe.pool.ntp.org), and the responder also mentioned that the NTP pool folks were working to fix the problem.Full thread here:
https://mailman.nanog.org/pipermail/nanog/2023-August/222706.html
Now the interesting part of that thread was that shortly after (~4 hours) the OP posted about their problem, someone replied and recommended that they setup their own GPS-based NTP network.From there, that sub-thread was mostly about NTP attacks and GPS security and mounting receivers on top of datacenter buildings and such.
Counting up the messages (if my mail client's threading capabilities are to be trusted), of the 94 or so total messages in the thread, 79 (84%) of the messages were in that GPS-related subthread, and only 15 messages responded to the OPs issue directly.
I didn't really see this as a failing of NANOG per-se, it's just a public mailing list and some folks went on a nerd sniping tangent (cf. XKCD #356).
A few years ago a team of researchers tried to use a feature attribute flag in BGP. Their experiment caused an important BGP software to crash.
The BGP community reacted by thanking the researchers for uncovering that flaw and worked on improving stability. No, just kidding. They shouted at the researchers and asked them to stop, which they eventually did.
I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers.
Sources: https://mailman.nanog.org/pipermail/nanog/2018-December/0984... https://mailman.nanog.org/pipermail/nanog/2019-January/09876... https://mailman.nanog.org/pipermail/nanog/2019-January/09918... https://mailman.nanog.org/pipermail/nanog/2019-January/09914...