This is why large phone networks have really long rollout periods - to hopefully catch these things before it affects everything.
This is also why emergency services should not be using cell phones for reliable service during emergencies.
Here's another example from 1990, where the phone network got stuck in a reset loop.
https://catless.ncl.ac.uk/Risks/9/62#subj2
Corporations make similar mistakes with their disaster planning. It doesn't matter if you purchase network connections from two suppliers if they go across the same bridge.
In the early 2000s as a cost cutting measure, the company decided to nix the radios for satellite phones on each truck. Of course this proved to be problematic, as the sat phones often had reception issues. They relied on cell phones as backups, which was also quite foolish as the remote compressor stations had very poor reception. They also had issues where someone would leave a voicemail and they wouldn’t get notified of it for days or weeks due to some issue with AT&T.
After a couple emergencies where communication was identified as a big issue, they proposed moving back to the old radio system, but they had already sold off the frequencies and dismantled the infrastructure. My dad retired not long after this, but the “corporate bean counter” trope rang quite true here, and in the long run we were all a little less safe because some executive with no field experience wanted to make a name for themself by saving a little money on something that proved to be mission critical.
It's an alarming trend among new grads as well, who see existing systems as bloated and in need of rewrites. Who would've thought the bloat was kind of important?
It's part of our culture that the young kids dismantle the 'old people stuff' setup their new stuff to only re-invent the wheel again and again, run into the same problems that were solved decades ago to only be repeated again.
Remember how TCL was going to save us all from K&R C? Then it was perl to save us from TCL? Then Ruby? Python? javascript, Go or whatever is the flavour of the month?
Unfortunately those bean counters probably got large raises and bonus. Then moved on to their next job proclaiming their victories and got out into higher positions.
Really sad results there, throwing away a sound investment & body of know how, for worse results everywhere, and less safety for everyone.
This is one of those cases where, if it isn't broke dont fix it was a rule to be followed.
Now its hard for the company to fallback to the old system, since they need to buy another set of frequencies, before they can even plan to rebuild the radio tower network.
Since they had stopped with the good running radio network to cut costs (unwise since it ran fantastic) the chance that they will invest in restarting such a network is not really big, unless they get heavy lossses due to their unreliable current system of modern (but worthless) gadgetry
I love the fact the "fix" was to reduce traffic load so that the issue was unlikely to occur and let things go back to normal.
> AT&T solved the problem by reducing the messaging load of the CCS7 network. That allowed the switches to rest themselves and the network to stabilize.
Doing real traffic engineering where shared risk link groups do not exist for the active and standby links is a lot of work and, moreover, is hard to keep working as operators change their networks (for maintenance, for other reasons).
On top of the basic difficulty, there are other issues - layering makes this opaque, the service providers have become completely de-skilled, the business model for MPLS is getting killed by Over-The-Top providers who pretend they can give you equivalent service using multiple links and IPSEC tunnels. And so on.
All of the unexpected shared infra issues I’ve observed have been down at this lower level that never shows up at the IP layer.
I worked at two small ISPs, each with a fiber network spanning a few US states. They had their own databases to track this information, but as far as I know there was no central database. One of them used OSPInsight backed by a self-hosted SQL Server database. I don't know what the other place used.
Because your workplaces don't rely on funding from taxpayers.
Right. Their workplaces probably had tighter and more transparent financial constraints with more oversight, more accountability for failure, and more incentive to do it right.
On a more serious note, I haven't seen any convincing evidence that your genetalizations hold true with reapect to public vs private organizations.
I have, and been a consumer of them, and have quit them easily.
> I haven't seen any convincing evidence [...]
I guess I could link a page of companies dissolving due to financial mismanagement vs governments or something. The point is not that only one sector suffers mismanagement, it's how that mismanagement is realized/handled/reacted-to, and of course it doesn't apply to every situation. I just get tired of hearing assumptions like a public service is improperly implemented due to funding, especially in this case where I know many of the municipalities are well endowed and should have the redundancy of their systems questioned. Why not ask for convincing evidence for those statements instead?
???
>Why not ask for convincing evidence for those statements instead
Because I'm not replying to someone making such a statement.
Meaning in many cases adverse effects of private sector failure are easier to divorce than public.
> Because I'm not replying to someone making such a statement.
Right, I was, hence the thread context. The point was wondering why replies aren't there asking, so of course the reply isn't there asking.
Government doesn't tend to provide "easy to divorce services", it's a selection bias problem, not something inherent to public vs privately ran organizations.
Of course it's easier to change office supply vendors than it is to change water companies. Utilities tend to be government operated (or heavily regulated) because their nature makes a competitive market difficult.
>Right, I was, hence the thread context.
Why did you agree with the person you were replying to then? You could have asked for proof in your reply, instead of affirming the comment?
Is this outage just breaking those places that didn't have such things (as a result of insufficient budget or skill)? Is there some now-regrettable contract(s) that prevented that from being in place?
Why is a system that saves lives working without the backup that systems that dont dictate life are? While I understand that govt has both budget and hiring issues, i've worked in state govt and there are definitely some highly competant admins to be found, so I wouldn't assume that is the problem without word from someone with more detailed knowledge into this particular problem.