News for hackers. Stuff that matters.
Re:Network Instability in NYC2 on July 29, 2014 by donavanm on Saturday August 02, 2014 @07:53PM (#8126484)
So an RE failed and flipped to the secondary. They run mlag from the agg layer to their tors. Said mlag had a grey failure. The time to recovery was predominantly fault detection. After detection they upgraded the os on the known good switch. Presumably this bounced, affecting remaining traffic. After upgrade they downed the failed switch, shifting traffic to the known good.

Moderately interesting that they run everything off a single agg pair. Also that they use mlag instead of routing/mpls/etc for availability.

Key finding is lack of visibility in to the layer 2 availability and performance. Would be interestin to see if they try to ecmp layer 3 or use existing lacp frames for fault detection in the future.

Re:Network Instability in NYC2 on July 29, 2014 by insaneirish on Saturday August 02, 2014 @08:51PM (#8126612)
They're probably learning that MLAG should be avoided unless absolutely necessary. But providing common L2 domains across cabinets is probably something they "need."

There's much less to go wrong with ECMP at L3. Stateful networking components frighten me.

Re:Network Instability in NYC2 on July 29, 2014 by donavanm on Monday August 04, 2014 @12:51AM (#8130378)
Not knowing anything about their infrastructure Id guess that they're using vlan tags for customer isolation. They'd want customer instances spread among racks, based on instance type etc. Going to in house/vxlan/nvgre encap certainly looks better suited, but still has a high bar to entry.
Re:Network Instability in NYC2 on July 29, 2014 by contingencies on Monday August 04, 2014 @04:25AM (#8130801)
You guys sound like you know a lot about these types of networking protocols. Can you recommend a decent summary of currently available approaches to relatively dynamic private link layer network topology provision suitable for cross-cabinet (or even cross-site) virtualized infrastructures and their drawbacks? For instance, I've been seeing Open vSwitch gaining popularity.