News for hackers. Stuff that matters.
Re:Grave flaws in BGP Error handling by hannob on Tuesday August 29, 2023 @09:13AM (#37307354)
Not surprised.

A few years ago a team of researchers tried to use a feature attribute flag in BGP. Their experiment caused an important BGP software to crash.

The BGP community reacted by thanking the researchers for uncovering that flaw and worked on improving stability. No, just kidding. They shouted at the researchers and asked them to stop, which they eventually did.

I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers.

Sources: https://mailman.nanog.org/pipermail/nanog/2018-December/0984... https://mailman.nanog.org/pipermail/nanog/2019-January/09876... https://mailman.nanog.org/pipermail/nanog/2019-January/09918... https://mailman.nanog.org/pipermail/nanog/2019-January/09914...

Re:Grave flaws in BGP Error handling by throw0101c on Tuesday August 29, 2023 @09:42AM (#37307794)
> I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers.

Or they felt like some of us feel about Tesla/Musk unleashing cars with "Full" "Self-Driving" on public roads.

Re:Grave flaws in BGP Error handling by adra on Tuesday August 29, 2023 @10:46AM (#37308731)
As lousy as these systems are, they kill a hell of a lot fewer people (for now) than drunks, so net positive for the time being.
Re:Grave flaws in BGP Error handling by hiddencost on Tuesday August 29, 2023 @11:41AM (#37309568)
You're confusing "musk killing people to compete for market share because he's woefully behind" with "driverless cars".

There are other people building driverless cars, who have safer systems that they rolled out reliably.

Musk isn't saving people from drunk drivers, he's trying to catch up with the competition and chose to kill people to catch up.

Re:Grave flaws in BGP Error handling by simiones on Tuesday August 29, 2023 @11:27AM (#37309343)
That only makes sense if all the people using self-driving cars would have driven drunk (or drugged, or sleepy) instead. Otherwise, you're replacing a decent driver with a sub-mediocre automatic system.
Re:Grave flaws in BGP Error handling by mschuster91 on Tuesday August 29, 2023 @10:51AM (#37308786)
The solution to drunk drivers is public transport, not self-driving cars.
Re:Grave flaws in BGP Error handling by zaroth on Tuesday August 29, 2023 @12:24PM (#37310232)
It’s 2023 and driving kills like 40,000 people a year.

We’ve been building trillions of dollars of public transit since about the same time as we started building the National highway system. So when exactly does public transit start saving all these lives?

A new train from SF to LA has been in the works since 1996. They are expecting partial completion for $30 billion by 2030 which is projected to carry ~15 million passengers * ~400 miles = 6 billion passenger miles per year.

The fatality rate for passenger cars is ~0.57 per 100 million miles, so eliminating 6 billion passenger miles could be reasonably expected to save about 60 * 0.57 = 34 lives a year out of the ~40,000 that are dying on the roads annually.

This is just rough napkin math showing the scale of the problem would require a roughly $30 trillion investment in public transit to appreciably shift passenger-miles away from passenger cars.

Public transit can be very useful. But it’s clearly not the solution to eliminating driving deaths.

Re:Grave flaws in BGP Error handling by autoexec on Tuesday August 29, 2023 @12:48PM (#37310565)
It seems unfair to say that public transport doesn't have a role in reducing deaths caused by drunk drivers just because the hugely flawed and half-assed public transport systems in our cities (which were designed to make public transport ineffective) haven't yet delivered on that promise. In places where cities and public transportation systems are well designed and useful people do use them and those places see far fewer deaths from drunk drivers than the US does as a result.

You argue that it would be very expensive to fix the problems we've made for ourselves in order to get useful public transportation and that's certainly true. It's also very expensive to invent self-driving cars and part that cost has already been paid with lives lost and human suffering, but that hasn't stopped us from developing those systems. We could decide to invest instead in improving our cities and infrastructure. We just choose to let obscenely rich people kill innocent Americans while they play with their profit generating spying car tech instead.

Re:Grave flaws in BGP Error handling by simiones on Wednesday August 30, 2023 @04:20AM (#37319087)
Haven't many cities actually dismantled public transport that they had already built (or rather, sold it to car companies "to operate more efficiently", who then proceeded to dismantle it so it didn't compete with their cars)?
Re:Grave flaws in BGP Error handling by zajio1am on Tuesday August 29, 2023 @08:54PM (#37316267)
Unfortunately at the time when drunken people are returning home from pubs/bars (e.g. 2 in the morning) most public transport do not drive or at least is significantly limited.
Re:Grave flaws in BGP Error handling by patmorgan23 on Tuesday August 29, 2023 @09:53PM (#37316696)
This is a problem with North Americas suboptimal implementations of public transit. Most public transportation systems in the rest of the world do actually take care of this.
Re:Grave flaws in BGP Error handling by yjftsjthsd-h on Tuesday August 29, 2023 @01:34PM (#37311197)
I think the difference is that network gear is supposed to handle the whole spec and gracefully deal with (drop) things that aren't to spec. If there was a law that all cars had to be able to get hit by another car and keep going without issue (and this was physically plausible), then it would be similar.
Re:Grave flaws in BGP Error handling by dogleash on Tuesday August 29, 2023 @09:37AM (#37307733)
> I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers.

I suspect a Network Operators mailing list is going to be full of people trying to keep production up, with the tools they've got, rather than the tools they'd like. Thinking the software quality on the routers they're using are dogshit is probably just called "Monday" and thinking their peers' routers are even worse is called "Tuesday".

I can understand they don't appreciate "research" done on production systems that might cause issues when they already know their equipment sucks.

Re:Grave flaws in BGP Error handling by themerone on Tuesday August 29, 2023 @10:08AM (#37308192)
Network operators are also the ones that pop into IPV6 discussions to voice their displeasure with having support another protocol.
Re:Grave flaws in BGP Error handling by booi on Tuesday August 29, 2023 @10:51AM (#37308788)
To be fair… supporting dual stack really does suck
Re:Grave flaws in BGP Error handling by yjftsjthsd-h on Tuesday August 29, 2023 @01:31PM (#37311146)
A lot about networking sucks, but that's what people are paying them for.
Re:Grave flaws in BGP Error handling by philg_jr on Tuesday August 29, 2023 @03:14PM (#37312459)
Also to be fair, IME there seems to be a lot more vocal pro-IPv6 people on the NANOG list than not.
Re:Grave flaws in BGP Error handling by kortilla on Wednesday August 30, 2023 @04:29AM (#37319137)
Not really, ISPs (who are on nanog) tend to like ipv6. The ones who complain are sysadmins and network operators of campus/corporate networks.
Re:Grave flaws in BGP Error handling by icedchai on Wednesday August 30, 2023 @06:29PM (#37329993)
They've only had over a decade to get their act together. I'm being generous, seeing IPv6 is older than the original IPv4 internet was when it went mainstream.
Re:Grave flaws in BGP Error handling by gnfargbl on Tuesday August 29, 2023 @12:44PM (#37310495)
I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own organisation is continuing to purchase equipment that sucks.

Bitching at researchers (note: not "researchers") who are finding and calling out those flaws before someone really bad comes along and exploits them is somewhere between unhelpful and negligent. Internet security is always improved by more sunlight.

Re:Grave flaws in BGP Error handling by netfortius on Tuesday August 29, 2023 @01:09PM (#37310844)
I suppose you have never seen, ever, in your professional career, the "C" suits making business deals and thus decisions on what's to be acquired, with big network/server/storage/database/software vendors, over the head and against the advice of those then called to support crap in production.
Re:Grave flaws in BGP Error handling by tw04 on Tuesday August 29, 2023 @01:43PM (#37311307)
>I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own organisation is continuing to purchase equipment that sucks.

I think you fail to realize that the internet is a bunch of disparate networks that no one person controls. When "fixing" a router means you break half the internet, you just don't fix it. This isn't some walled garden where you can just say "tough shit if you broke, fix it".

Most of the "issues" these researchers find are well known, but backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.

Re:Grave flaws in BGP Error handling by gnfargbl on Tuesday August 29, 2023 @02:03PM (#37311579)
> I think you fail to realize that the internet is a bunch of disparate networks that no one person controls.

The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed. Once a vulnerability is known then someone out there is going to start exploiting it almost immediately.

> backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.

Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP. This is a security lesson that has been learnt in recent years by many sectors of the IT industry and, judging by the woeful response that Ben Cartwright-Cox describes in his post, it's one that the network backbone is going to be learning soon.

Re:Grave flaws in BGP Error handling by tw04 on Tuesday August 29, 2023 @02:24PM (#37311828)
>The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed.

So you've never worked in a corporate environment. Here's how that conversation would go:

Hey guys, some researchers found out if you use a test flag meant for the lab on the public internet, it breaks all our BGP sessions.

OK, so drop their feed and block them?

Great, done.

>Once a vulnerability is known then someone out there is going to start exploiting it almost immediately.

And yet the "vulnerability" in question was known, and was not being immediately exploited if you read through the mailing list or were participating in NANOG at that point in time. So your statement is provably false.

>Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP.

And yet we're having this conversation in 2023, they have operated the same way for 40+ years, and somehow the internet is still working. Bad actors get blackholed, it worked in the past, it'll continue working in the future. The reality is that backbone routing is expensive, and expecting everyone to update their kit on YOUR timeline isn't reasonable.

Re:Grave flaws in BGP Error handling by gnfargbl on Tuesday August 29, 2023 @02:45PM (#37312105)
I am aware that network operators have behaved this way for very many years. The point I am making is that all of the IT industry used to attempt to "work around" security vulnerabilities in the same way, until log4shell and all the others gradually beat that propensity out of them.

I am prophesying that a similar reckoning is likely to come upon the network backbone. You're arguing that the cost of entry to the game (BGP peering) is such that the old ways will continue to work. Let's hope you're right.

Re:Grave flaws in BGP Error handling by devonkim on Tuesday August 29, 2023 @06:55PM (#37315240)
Software ecosystems like libraries and frameworks have completely different propagation and remediation mechanics compared to federated systems like core Internet backbone routers and switches is the thing. Try as we might conceptualize otherwise the modern Internet from a packet’s purview is more like a loose confederation of ultimately privatized or state-run fiefdoms than a cellular automata digraph explosion. So actors that try to act maliciously against the network will be basically shut out given the rule of an iron fist being the default.
Re:Grave flaws in BGP Error handling by gnfargbl on Wednesday August 30, 2023 @05:06AM (#37319333)
Conceptually, there is no special difference between a remotely exploitable vulnerability in an HTTP server and a remotely exploitable vulnerability in a BGP router. It would have been theoretically possible to have dealt with log4shell by blocking every IP address that was sending malicious payloads, for instance.

Practically, I accept that there is a real difference. ASNs are much harder to acquire than IP addresses, and there are far fewer of them. That difference might mean that blocking continues to be an effective mechanism. However, in a world where malicious actors are increasingly just nation states wearing a flimsy mask, I'm not sure that the difficulty of getting access to sufficient numbers of ASNs is going to be a forever mitigation.

The main thing I have taken away from this thread is that when it comes to vulnerability management, network operators seem to be pretty far behind the curve in mindset terms. The arguments being made in here ("you can't break other people's stuff", "we have a lot of boxes and SLAs mean we can't patch quickly") aren't even slightly new. The well-proven reality is that threat actors don't give one single shit about any of that.

Re:Grave flaws in BGP Error handling by eskaytwo on Wednesday August 30, 2023 @05:08AM (#37319352)
Operators could have raised this with vendors to fix.
Re:Grave flaws in BGP Error handling by robertlagrant on Tuesday August 29, 2023 @03:42PM (#37312861)
> OK, so drop their feed and block them?

> Great, done.

The "great" is the problem, no?

Re:Grave flaws in BGP Error handling by hyperionplays on Tuesday August 29, 2023 @11:15PM (#37317240)
Backbone operator who was effected by this. We had a large number of routers in production with this bug, we were aware and upgrading as fast as we could, but with 99.999% uptime SLA's we only have so many minutes per router we could afford for downtime/outages. We had schedules in place (approx 3 months of out-of-hours upgrades) 1 week warning was a bullshit move. Dropping the BGP sessions on 1000's of routers globally was stupidity.

Bearing in mind we also couldn't just "apt-get upgrade" in place, most boxes required hard reboots to apply the patches.

The answer to your question is no, as per a few others have said, Don't violate Rule #1. We see bad actors very often, our job is to keep the bits flowing and the internet online.

Keeping the internet online is painful enough as it is without "researchers" dropping thousands of routers to "prove a point."

Re:Grave flaws in BGP Error handling by SSLy on Wednesday August 30, 2023 @04:59AM (#37319295)
Verbatim this is the same things people said in early 00's about people testing XSS et al. against poorly coded PHP/Perl sites.
Re:Grave flaws in BGP Error handling by zamadatix on Tuesday August 29, 2023 @06:09PM (#37314798)
The internet does not owe a negligent operator peering. No one person controls the internet but you sure as hell can find yourself kicked out if your goal is to be right on principals instead of a good peer. The only hubris which could be at play is someone thinking the rest of the internet should follow their desires instead of the other way around.

Research is great but if you're poking around breaking things via DoS in the internet you're going to be judged as a bad peer regardless what the route daemon does. The security of the internet constantly moves forward but that doesn't mean the expectation of improving has to hold more precedence than the expectation of running. It's still very much a human to human system for all but the actual route exchange.

Re:Grave flaws in BGP Error handling by hn8305823 on Tuesday August 29, 2023 @02:00PM (#37311534)
> I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks

We call out vendors when their quality suffers but you only have two or three practical vendor options for high end routers depending on your use case/feature requirements. All three have varying cycles of hardware and software quality, with hardware and software rarely being in phase for the same vendor.

The fact that all three vendors make most of their money on non-network focused enterprises with less sophisticated engineers doesn't help get development attention where it needs to be for global infrastructure.

Re:Grave flaws in BGP Error handling by ketralnis on Tuesday August 29, 2023 @07:22PM (#37315512)
I practised picking locks on your house and broke the lock because of a poor lock design. Yeah okay the lock sucks. But I probably shouldn't have been doing it on your house in the first place and now you've got a broken lock.
Re:Grave flaws in BGP Error handling by timnetworks on Tuesday August 29, 2023 @11:35PM (#37317365)
"I can get into this lock" will not have the same effect or call to action as "I can get into your doors" but it admittedly has to be aimed at the landlords, not the doormen.
Re:Grave flaws in BGP Error handling by zajio1am on Wednesday August 30, 2023 @09:05AM (#37321481)
That is not really a matching comparison. With BGP, there is kind of public space (global BGP tables) and participants are free to attach valid attributes to their routes. From a cursory look, these researchers used announcement that was compliant with BGP standards (in contrast to Ben Cartwright-Cox from this article, who used fuzzed invalid attributes), so they did something that was generally acceptable behavior for BGP participants.
Re:Grave flaws in BGP Error handling by patmorgan23 on Tuesday August 29, 2023 @09:56PM (#37316726)
Rule number 1 of the internet is don't break other people's shit. You can do whatever you want on your own equipment but don't break other people.
Re:Grave flaws in BGP Error handling by loa_in_ on Wednesday August 30, 2023 @11:35AM (#37323764)
Except there's no "staging Internet" to test such equipment, and even if there was it would still sometimes not translate to the real open network. I'm sorry, were they inconvinienced with the world at large?
Re:Grave flaws in BGP Error handling by capableweb on Tuesday August 29, 2023 @09:44AM (#37307830)
Is there a reason why the researchers did this on live production BGP network rather than in a test environment, running their own BGP and FRR routers? Seems a bit haphazardly, as doing so first seems to have uncovered the issue.

Overall I agree with your message though, and of course things shouldn't stop working just because you use attributes reserved for development.

Re:Grave flaws in BGP Error handling by LegionMammal978 on Tuesday August 29, 2023 @09:59AM (#37308069)
The researchers claimed that they had first tested it with a few different vendors' routers without any issues, but they had not included FRR in those tests [0]. There was a bit of discussion on the mailing list regarding just how many BGP implementations (and configurations) a responsible researcher is obligated to test with before exposing something new on the public Internet.

[0] https://mailman.nanog.org/pipermail/nanog/2019-January/09876...

Re:Grave flaws in BGP Error handling by salawat on Tuesday August 29, 2023 @02:18PM (#37311759)
The correct answer is "All of them on your own network first".

Thou shalt not blow up another's network knowingly.

Blowing up another's network because you didn't test your theory in your own isolated lab space first counts as blowing up another's network knowingly. Because what the hell else did you think would happen?

Cripes.

Re:Grave flaws in BGP Error handling by ExoticPearTree on Tuesday August 29, 2023 @10:41AM (#37308667)
The sane thing to do is to drop that particular announcement with that invalid attribute or discard the invalid information. But some people tend to be more catholic than the pope.

Dropping the whole session, especially in a DFZ scenario is really bad. The churn that you generate is immense for all your neighbors.

Re:Grave flaws in BGP Error handling by fanf2 on Tuesday August 29, 2023 @11:02AM (#37308968)
As the fine article explains, that is what RFC 7606 specifies.
Re:Grave flaws in BGP Error handling by ExoticPearTree on Wednesday August 30, 2023 @03:13AM (#37318660)
Yeah, but I don't think the people who created that RFC thought about all the bad side effects their advice causes.

BGP is one of the most robust protocols out there: there's no state (as in protocol state), you can treat each message independently, you can extend it super easy and so on.

And because you can treat every message independently, you can have a really secure and stable mechanism to deal with any badness that might happen.

Re:Grave flaws in BGP Error handling by crest on Tuesday August 29, 2023 @11:33AM (#37309441)
Just have a bunch of fuzzers running 24/7 to keep all implementations honest.
Re:Grave flaws in BGP Error handling by clankyclanker on Tuesday August 29, 2023 @12:44PM (#37310498)
That’s what Google used to do (though not for BGP), back in their engineering days. That’s what gave us AFL.
Re:Grave flaws in BGP Error handling by tialaramex on Tuesday August 29, 2023 @01:18PM (#37310979)
And indeed GREASE the TLS feature where you just propose nonsense because every other party should go "Er, no I don't speak nonsense?" to prevent ossification also comes from Google engineers.

GREASE means that when you build your "advanced" security device which "protects" customers by treating everything different as hostile, it won't work, so you'll need to tweak it to at least just ignore such differences as irrelevant, which is enough that we can come back later and intentionally improve things.

Previously, without GREASE, we'd have to guess what oversights we could exploit to avoid this "protection" to deliver protocol improvements, if we guessed wrong nothing works, or everybody's security is broken, sometimes both. e.g. for TLS 1.2 the oversight we found is, if you're resuming an existing session the "security" products just wave that through because otherwise they break people's real workflows. So in TLS 1.3 protocol as actually spoken essentially an initial connection goes like this:

Client: Hi some.dns.name.example I'm a TLS 1.2 Client, I'd like to resume our previous conversation #randomNonsense. Also, completely unrelated, I happen to speak FlyCasualThisIsReallyTLSv1.3 and so I have these TLSv1.3KeyAgreementParameters.

then either:

TLS 1.3 Server: Hi Client, of course, let's continue from there. [ whereupon everything further is encrypted because this is actually TLS 1.3, but to a dumb middlebox it makes sense that they're just resuming a prior encrypted conversation #randomNonsense which it doesn't remember ]

OR

TLS 1.2 Server: Er, no I don't remember any such conversation and I don't know FlyCasualThisIsReallyTLSv1.3 so let's start a fresh conversation as normal.

Re:Grave flaws in BGP Error handling by hinkley on Tuesday August 29, 2023 @12:44PM (#37310499)
I think you’ve discovered the secret to getting people to exercise more.
Re:Grave flaws in BGP Error handling by salawat on Tuesday August 29, 2023 @02:24PM (#37311827)
More like the secret to getting ignored/blackholed/sued into the ground the moment your fuzzer hits on a combo that actually does damage to someone else's system, since the implication here is you're not doing it on your own stuff.

If you are, good on ya. Continue doing $deity's/Bob's work.

Re:Grave flaws in BGP Error handling by pixl97 on Tuesday August 29, 2023 @02:32PM (#37311938)
More like the secret is to buy up some routing equipment and run these tests on your own network, sell the exploits you find to nefarious actors for big bucks, and have a bunker to survive the crash of the internet in.

If your system is on the internet it is now a potential piece in a potential global war.

Re:Grave flaws in BGP Error handling by salawat on Tuesday August 29, 2023 @02:46PM (#37312113)
Always has been, even since it's inception. The only thing that'd kept it safe and unutilized in that regard had been a grassroots efforts to not facilitate that transformation. That only lasted so long as the main population of Ops/devs/netizens were in agreement.

We've blasted past that age at this point, do everybody gets to don their robe and wizard hat, pick up the staff, and yeet patches, routing rules, and packets betwixt one another.

Re:Grave flaws in BGP Error handling by hinkley on Tuesday August 29, 2023 @12:42PM (#37310471)
Recipe as I got it from grandma

1) Beleaguered people misbehave.

2) You are usually the architect of your own destruction.

3) It’s easier to see other people’s problems than your own.

Stir into a death spiral and let it rest for six to twelve months. Bake at 375° for two years, then serve cold with a sprinkling of schadenfreude to offset the bitterness.

Re:Grave flaws in BGP Error handling by braiamp on Tuesday August 29, 2023 @01:31PM (#37311154)
I don't think that is an accurate representation of the "comunity", it was a "singular complaint from a company advertising unallocated ASN and IPv4 resources", and got told off by literally most of the mailing list. I'm counting at least 20 other individual responses and all of them are in support of the experiment https://mailman.nanog.org/pipermail/nanog/2019-January/threa...
Re:Grave flaws in BGP Error handling by tw04 on Tuesday August 29, 2023 @01:48PM (#37311392)
If you read the links: they discovered that using this flag broke EVERY installation of FRR on the internet. They let the developers know. And then said they were going to test again in a week.

Does any sane person think that 1 week is enough time to both notify every user of FRR in the WORLD, as well as ensure they have upgraded their installations for a "bug" that only is triggered by someone using experimental flags meant for testing purposes only?

Mind you, these researchers didn't test ANYTHING in a lab environment first "because it's a lot of work to test all the open source BGP implementations in the wild". AKA: I'm lazy.

Re:Grave flaws in BGP Error handling by LegionMammal978 on Tuesday August 29, 2023 @05:43PM (#37314533)
> Mind you, these researchers didn't test ANYTHING in a lab environment first "because it's a lot of work to test all the open source BGP implementations in the wild". AKA: I'm lazy.

That's a gross exaggeration: they said that they first tried it "in a controlled environment with different versions of Cisco, BIRD, and Quagga routers" [0] without issue. Presumably, the problem we know about would have been avoided if they had happened to include FRR in their test suite beforehand. But that raises the next question, exactly how many implementations does one need to test before one is no longer "lazy"?

[0] https://mailman.nanog.org/pipermail/nanog/2019-January/09876...

Re:Grave flaws in BGP Error handling by xyst on Tuesday August 29, 2023 @01:52PM (#37311444)
To be honest, if you are a responsible researcher you are NOT running “experiments” in live environments that have potential to impact multiple regions.

The outrage here is understandable. I would be pissed if I was paged because halfway across the world, some asshole caused this intentionally. Also this mailing list is aimed at the operators, not the vendors. Different communities.

The vendor response here, I do agree is irresponsible. Besides OpenBSD, every one here just sucked.

Re:Grave flaws in BGP Error handling by pixl97 on Tuesday August 29, 2023 @02:27PM (#37311870)
Ya the researcher could just sell their findings to whatever blackhat groups are out there instead. You generally get less pushback and more cash from them.
Re:Grave flaws in BGP Error handling by kjartann on Wednesday August 30, 2023 @10:50AM (#37323022)
Two thoughts:

- I think it's OK not to want experiments like these to be conducted connected to the global internet and rather in a lab such as the one benjojo used.

- NANOG is known to be a bit of a joke within the networking space, not too long ago someone complained about NTP issues with traffic passing through TATA and they were told to make their devices stratum 0 or something to that effect, so don't always take them too seriously.

I think the first applies here.

Re:Grave flaws in BGP Error handling by njt on Wednesday August 30, 2023 @03:17PM (#37327499)
> someone complained about NTP issues with traffic passing through TATA and they were told to make their devices stratum 0 or something

Not saying your summary of that thread is wrong, I just wanted to expand a bit to maybe clarify:

    - someone posted to NANOG about issues with their (European) servers reaching 0.freebsd.pool.ntp.org
    - they noticed they were being routed through Tata (AS6453)
    - they mentioned that their other servers (e.g. in Africa) had no issues
    - they posted to NANOG to ask about this
    - they thought it was a routing or FreeBSD issue (they later posted about the issue to a FreeBSD list as well)
So AFAICT, the correct response was posted about a day and a half (~42 hours) later:

    - the NTP pool uses GeoDNS to map clients to servers
    - the OP's server IP address was geolocated to Mauritius
    - the Mauritius zone in the pool has only one server
    - so the OP would only ever get that IP as an answer
The short-term fix was to use a different pool address (e.g. europe.pool.ntp.org), and the responder also mentioned that the NTP pool folks were working to fix the problem.

Full thread here:

    https://mailman.nanog.org/pipermail/nanog/2023-August/222706.html
Now the interesting part of that thread was that shortly after (~4 hours) the OP posted about their problem, someone replied and recommended that they setup their own GPS-based NTP network.

From there, that sub-thread was mostly about NTP attacks and GPS security and mounting receivers on top of datacenter buildings and such.

Counting up the messages (if my mail client's threading capabilities are to be trusted), of the 94 or so total messages in the thread, 79 (84%) of the messages were in that GPS-related subthread, and only 15 messages responded to the OPs issue directly.

I didn't really see this as a failing of NANOG per-se, it's just a public mailing list and some folks went on a nerd sniping tangent (cf. XKCD #356).