Would’ve been great to come out of an event like this looking at whether catastrophic fault conditions like this exist elsewhere in our national infra, but it feels like all that’ll happen is Optus gets the shit kicked out of them while other providers count their blessings
It's that they took so long to restore. Carrier systems are supposed to be multiply fault tolerant.
Even if you have to put warm bodies physically in front of them, you should have enough of those people around to get the core networks up and going again within an hour or so by rolling back to a known-good configuration.
Even if they've somehow managed to brick the systems, they should have enough hot/cold spares, and the ability to call $VENDOR to hand-deliver new ones if need be.
The employees locally lack the knowledge and the access to restore the network without guidance from external vendors outside the country. From an operating cost perspective, this is the right choice, but for reliability and sovereignty it's terrible.
Sadly, with RCS, 5G Standalone and other new technologies that require operating servers with leading edge software, operators repeatedly choose to outsource the entire stack to an external vendor like Nokia rather than replicate that knowledge locally.
Voice calls should be prioritized over other network traffic. Data should be the same. And my phone should roam from AT&T to VZW (or even TMO).
[1] having said that, I've not looked too closely, and it sounds like (unsurprisingly) that might not exactly be the reality...
- The CEO told the senate hearing that she now carried Telstra and Vodaphone SIM cards (maybe practical, terrible optics)
- Transport for NSW is heavily reliant on Optus so public transport was heavily impacted (not entirely Optus' fault, but public outrage means someone had to be the scapegoat)