News for hackers. Stuff that matters.
Re:DE-Cix – Power failure leads to Downtime in germany by virtuallynathan on Monday April 09, 2018 @06:09PM (#16797417)
Kinda fun to see the increase in AMS-IX traffic: https://ams-ix.net/technical/statistics

Keep in mind this is only one of the many DE-CIX datacenters, they exist in at least 8 datacenters in Frankfurt: https://twitter.com/goebelmeier/status/983453047543214081

Only those peers who are single-homed to Interxion FRA5 / DE-CIX6 will have experienced an outage.

Re:DE-Cix – Power failure leads to Downtime in germany by AdamJacobMuller on Monday April 09, 2018 @10:41PM (#16798622)
In fairness, we're in FRA8 but DE-CIX is not directly, so we're connected to their switches in FRA5.
Re:DE-Cix – Power failure leads to Downtime in germany by usernam33 on Monday April 09, 2018 @05:41PM (#16797241)
Imagine it being 8pm, you sit in a operating room of germanys largest internet knot with up to 6,027 Tbit/s throughput and then the power goes down.

I am concerned that this even happened in a place important like this.

pic related: http://img.pr0gramm.com/2018/04/09/9286baa4b2261184.jpg

The article states internet was down for users at around 10pm (I expirenced it live). So they might have been on generators for two hours (speculation).

Shouldn't we decentralize more?

Re:DE-Cix – Power failure leads to Downtime in germany by virtuallynathan on Monday April 09, 2018 @06:09PM (#16797420)
Only those peers who are single-homed to Interxion FRA5 / DE-CIX6 will have experienced an outage. The whole of DE-CIX isn't down.
Re:DE-Cix – Power failure leads to Downtime in germany by AdamJacobMuller on Monday April 09, 2018 @10:47PM (#16798644)
> Shouldn't we decentralize more?

> The whole of DE-CIX isn't down.

Largely agreed with you, though as general advice I would say that redundancy by adding more exchanges to your network serves you better than adding diverse connections to a single exchange.

Re:DE-Cix – Power failure leads to Downtime in germany by virtuallynathan on Tuesday April 10, 2018 @10:53AM (#16801782)
Yep, that makes a lot of sense, especially since 10G/100G waves and such on the FLAP route are so inexpensive. Getting a remote peering port on AMS-IX or something is probably a good move.
Re:DE-Cix – Power failure leads to Downtime in germany by mmarx on Monday April 09, 2018 @06:57PM (#16797709)
> The article states internet was down for users at around 10pm (I expirenced it live). So they might have been on generators for two hours (speculation).

More likely, the timestamp in the picture is UTC, which corresponds to 22:00 CEST, cf. [0]

[0] http://status.core-backbone.com/index.php?id=1972

Re:DE-Cix – Power failure leads to Downtime in germany by walrus01 on Tuesday April 10, 2018 @12:27AM (#16798986)
Well, now we know that significant portion of that facility are 1+0 for power systems. In a true 1+1 setup with air gap between A and B side power systems this wouldn't have happened.
Re:DE-Cix – Power failure leads to Downtime in germany by AdamJacobMuller on Monday April 09, 2018 @10:40PM (#16798620)
We were hit by this.

In fact, we're hit by this again, down again at 22:09:19 EST for me and still down. Their website is broken too (https://www.de-cix.net/) and their email servers are offline which makes contacting them amusing.

Not much to be done, really, fortunately the impact for us is very trivial.

Re:DE-Cix – Power failure leads to Downtime in germany by avh02 on Tuesday April 10, 2018 @04:59AM (#16799999)
General home connectivity also suffered, at least a few people (and myself) had connectivity issues, if you search for @vodafone_de on twitter you'll see a lot of people complaining (though I assume it was not just vodafone with issues)
Re:DE-Cix – Power failure leads to Downtime in germany by mntnman on Monday April 09, 2018 @11:05PM (#16798710)
Just received an email update from INTERXION acknowledging they did indeed lose power again at FRA5. This leaves us in a bit of a mess.