As someone who understands networking at the office / rack level (but not the datacenter or nation-wide level), what are the single points of failure that could bring down an entire nationwide network? I would think that things would be setup by regions just to avoid this type of scenario.

At least the 3G networks are still up for voice/sms fail-over.

At a guess, things like nation wide authentication servers? (I don't know how devices are authenticated in LTE though)

That is a good guess - the Home Subscriber Server is the authentication server in an LTE network. Short of this or a PCRF (Policy Server), I cannot think of any other single element that could bring the network down. Routing (as identified by another reply) could be an issue, but I would assume that 3G and 4G data use the same core network.

I have a feeling it might be the routing to the hss/pcrf that could have been the isue. Almost every carrier has redundancy for these products and they are often in geographically distinct areas. Perhaps the routing from the eNodeBs/edge network to the core network was handled by another division within Verizon or even leased from a third party and could have caused all of this

Or a disgruntled employee left a cron job on all the key systems to rm -rf :-)

Things are usually not properly tested, you have to deal with a lot of interoperability issues from different vendors which only get played out on carrier network, logic fails by employees/integrators such as assigning an ip addresss in one corner of the nation and re-using it elsewhere inadvertently due to poor IP address managment, errors flooding a system which cause it to self-reboot which then signals to other systems to also self-reboot or simply a core central node such as hlr/hss has failed and proper geo-redundancy was not implemented

pquerna's guess is a reasonable one, though I don't know the authentication mechanisms, either.

Here are some other wild guesses that may have little or no resemblance to what Verizon is experiencing, but kind of show what can bring down an Internet-like network:

* Some sort of configuration fat-finger that got rolled out everywhere without proper testing. You'd like to think that's impossible on this scale, but after having an insider's look at the operations of a couple major US cable companies, I put nothing past telecoms.

* Verizon probably runs some form of routing protocol internally. If a core router has gone insane, you could, conceivably, get route flapping and bring down pretty much everything, but that's not supposed to happen on carrier-grade equipment (cough see previous paragraph).

* In a closely related scenario, large portions of the public Internet have been brought to their knees at least once by a nasty BGP bug in a major vendor's routers that got triggered when somebody broadcast just the right set of options in their announcement. Picture a guy in upstate New York trying to route around a bad line card, and suddenly the US goes dark because half the routers crash when they got the BGP table update.

You can probably spot a common theme here -- a "decentralized" but homogenous network is still vulnerable to the triggering of problems shared by all points.

The thing you have to understand about "wireless" systems like CDMA/GSM/HSPDA/LTE is that there is little diversity in the vendors of these types of systems and there are a bunch of relatively closed protocols that govern how things like the base stations authenticate subs and do provisioning of stations. It's not like IP; it's more like Novell IPX.

So even though LTE is ostensibly an IP-based system, there are a host of dependencies on single vendor closed management systems and protocols that could be the cause of any outage. A nationwide outage points to the failure of one of these systems to properly handle letting subscriber stations onto the network, not something simple like getting packets from here to there.

Think failure of the RADIUS/DNS like systems, not the routing infrastructure.

Your premise is more accurate, you just need to replace IPX with 3GPP standard protocols (vendors dont make their own protocols - everything is standardized), and instead of radius/dns (which are used in telcos nonetheless), its more likley HLRs,CSCFs, MMEs, or PDN gateways that can bring an entire network to its knees, or a poorly implemented mobile backhaul solution

Very good points. It would be interesting to hear (technically) from Verizon about the issues and resolution.

The connection between the network and Langely, through which all data passes.

As a current 4G customer, I can say this has been the most frustrating 6 months. It's been well documented their LTE network has been sketchy from the start:

4G launched in Dec 7th 2010 and this article ran a week later about issues already showing up with handoffs between networks: http://bit.ly/td1XQT

Another article from April 29 2011 about an earlier 4G outage: http://bit.ly/v8zJUZ

I'm not surprised this happened and will be returning to 3G. Clearly their 4G is not ready for prime time yet.

I'm not too familiar with 4G tech - but why "move back" to 3G? If the LTE network falls over, your phone switches to 3G and keeps right on humming along more slowly, doesn't it?

Or is my concept of 4G modems out of date?

I'm actually surprised at how resilient the cell phone networks are. For something that is so heavily used I would expect it to go down more often.

Or maybe my definition of going down is different than others. There are plenty of bugs all the time (dropped calls, misdialed numbers, etc). I'm not convinced that whenever I dial a number and someone else picks up and says wrong number that I did indeed dial the wrong number.

Ghost in the shell I tell you.

> There are plenty of bugs all the time (dropped calls, misdialed numbers, etc).

Precisely. Erlang was created to run digital phone switches, and so was designed exactly for the scenario of "it's fine if certain calls crash, they can always be redialed, but the system itself always has to be available".

that's interesting. a fake telecom spec (4G!) can't perform like a 3G network.

Care to elaborate ? Are you referring to the fact that 4G was meant to be 1 Gbps but marketing departments hijacked the term to just mean "as fast or faster then 3g"

yea pretty much. the whole term, 4G, is just a dubious marketing term meant to mislead consumers.

The $299 price tag for a Verizon Galaxy Nexus is looking like an even worse deal now.

4G is down: 23 customers affected.

I know it's an overly snarky comment, but being from a state that has literally ZERO 4g coverage, I gotta share the frustration. You have to live in a huge city to even get the signal, and when you do get it, your battery consumption is greatly increased. 4G as it stands for me is a huge pile of care not.

You actually don't have to live in _that_ large of a city. The Raleigh/Durham MSAs combine for about 1.6M in total population spread over a pretty sprawling area and we have had LTE service for a few months.

I frankly don't get the complaints or the folks who are going to downgrade to 3G. If you don't have LTE available, the device will fall back to 3G. When you do have LTE available, however, the performance can be unbelievable. I was co-working with a colleague and we were getting 20Mbps down and 10Mbps up over a little MiFi.

I'll hang onto my LTE MiFi, thankyouverymuch. :)

For comparison, the best I have measured with a 3G technology (HSDPA) was 10 Mbps down, while downloading from kernel.org on my laptop, tethered to my phone over wifi, in a sparsely populated area covered by a strong HSDPA signal.

I don't live in any large city. I'm 30 miles from the nearest one, Philadelphia, and have 4G service on Sprint. I've been on 4G since 6/2010.

(In the voice of the Apple fan in the … Samsung? LG? HTC? commercial: Why don’t you just get a 4G phone?)