BT fault hits broadband users and banks

Re:BT fault hits broadband users and banks by thomseddon on Wednesday July 20, 2016 @06:44AM (#12128030)

It's a power outage in Telecity Harbour Exchange, from the LINX mailing list this morning:

08:12

- We are currently experiencing an outage impacting some of our equipment located at Telecity/Equinix Harbour Exchange in London. It is impacting both the Juniper and Extreme LANs.

- We are still investigating to get further information, but we have lost all management access to the devices at the moment.

08:45

- We have had confirmation from Telecity/Equinix that there was indeed a partial loss of power impacting one of the power feeds.

- Power failed at 07:55 BST and was restored at 08:17 BST for the majority of our equipment.

- Unfortunately one of our edge routers (edge4-tch) on the Juniper LAN is still down at the moment.

09:33

- Power to edge4-tch has also been restored now. The router has been back up since 9:15 BST.

- All member services should be restored.

- We have an engineer on site and are working with Telecity/Equinix to understand why we lost all power to that device.

- We are also waiting for a full report from Telecity/Equinix about the cause of the power failure.

---

We don't peer with BT so ou traffic was OK but we did notice loss of service from AWS to BTNet

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by mattlondon on Wednesday July 20, 2016 @07:14AM (#12128139)

Great update thanks.

Perhaps I am being naive, but I thought data centers had back-up power to cover this exact problem? I.e. battery power until the generators kick in... or is this how it used to be done and these days no one bothers and just relies on fail-over to other sites????

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by pyvpx on Wednesday July 20, 2016 @07:21AM (#12128169)

despite best efforts, sometimes the generators don't kick in. sometimes not all of them kick in. sometimes one ATS in one feed in one section that just happens to house one half of an important switch chassis doesn't work as it did during tests last quarter.

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by snowy on Wednesday July 20, 2016 @07:51AM (#12128281)

True. But Data centers usually have an A and B power systems independent of each other for redundancy . Equipment is usually connected to both. The failure of either should not cause an outage. (Unless some one is stupid enough to connect core network equipment to only A or B power)

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by marcusr on Wednesday July 20, 2016 @07:58AM (#12128305)

Sometimes only A power or B power is delivered to the rack (otherwise you could receive different phases).

Power failures also occur sometimes when the issue is on the wrong side of where power is distributed - we've lost power to our servers when maintenance has gone wrong for example.

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by snowy on Wednesday July 20, 2016 @08:04AM (#12128323)

From my point of view if you end up with mission critical equipment connected only to A or B power then its engineered wrong.

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by edwhitesell on Wednesday July 20, 2016 @08:44AM (#12128544)

Very true, but sometimes this happens at fault of the facility, not the affected company. I've been in a facility experiencing widespread power issues because customers were incorrectly delivered A+A or B+B power, instead of A+B.

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by Aloha on Wednesday July 20, 2016 @02:38PM (#12131386)

This is why I'm a proponent battery floated of DC Power for mission critical systems.

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by chrishacken on Wednesday July 20, 2016 @09:34AM (#12128859)

Exactly. You can only test for these scenarios to a certain extent and even when everything works as expected during tests, things still tend to break during production when shit hits the fan.

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by pbhjpbhj on Wednesday July 20, 2016 @08:01AM (#12128314)

Like you, I'm surprised.

I thought major data centres had redundant power from the grid too (not just battery and generator), so the UPS smooths the power transition as the switching gear fails over to the alternate grid connection; then generators come up if that fails too? Is that considered over-engineering?

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by thomseddon on Wednesday July 20, 2016 @08:40AM (#12128511)

They usually do, but these only protect from grid failures. HEX certainly does have battery + generators (it's one of the most important DC's in the country).

Components inside the DC should be redundant, but the customer can decide whether to have dual power feeds from redundant areas of the DC provided to their rack/cage.

Even with all this, issues do happen, Telecity had a very high profile and prolonged issue at the end of last year that affected AWS & Azure amongst others: http://www.theregister.co.uk/2015/11/18/telecity_outage_fix_...

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by redcalx on Wednesday July 20, 2016 @08:18AM (#12128385)

Perhaps a more serious point is that a single power failure (in one data center) can have such a wide impact.

[ Parent | on HN ]

Re:BT fault hits broadband users and banks by voltagex_ on Wednesday July 20, 2016 @09:45AM (#12128923)

[ Parent | on HN ]