News for hackers. Stuff that matters.
Re:AT&T Fiber in the SF Bay Area is flipping bits by myself248 on Tuesday December 08, 2020 @03:31AM (#25343148)
They passed testing because they had each been individually crossconnected to a test trunk, and test calls force-routed over that trunk. Then to place them in service, the crossconnects were reconfigured to place them at their normal location in the system. The testing was to prove the voice path of each DSP card, and that those cards were wired into the crossconnect properly.

All that was true, the failure happened when it was being taken out of testing config and into operational config. Either nobody considered that that portion could fail, or the urgency to add capacity to a suddenly-overloaded network meant that some corners were cut. (Marketing moves faster than purchasing-engineering-installation...)

Oh, and as to the point about keeping the call path ephemera. Yeah probably, but in a server context, that'd be akin to logging the loader and MMU details of how every process executable is loaded and mapped. Sure, it might help you narrow down a failing SDRAM chip, but the other 99.99999% of the time when that's not the problem, it's just an extra deluge of data.

Re:AT&T Fiber in the SF Bay Area is flipping bits by jcims on Tuesday December 08, 2020 @04:27AM (#25343420)
Were the cross-connected circuits channelized or individual voice calls (ds0? I can’t remember from my wan days) or something else?
Re:AT&T Fiber in the SF Bay Area is flipping bits by myself248 on Tuesday December 08, 2020 @04:58AM (#25343577)
As I recall, the cross-connects were done at the DS1 level, and an individual card handled 24 calls. These are hazy, hazy memories now; this took place in 2004-ish.
Re:AT&T Fiber in the SF Bay Area is flipping bits by jcims on Tuesday December 08, 2020 @05:14AM (#25343635)
Nice!! Thanks for the walk down memory lane, this was cool.