Re:Network Instability in NYC2 on July 29, 2014
by imbriaco on Saturday August 02, 2014 @10:51PM
(#8126867)
We have very competent network engineers, but it never hurts to have a second opinion. Verifying that there are no subtle interactions that we've missed is simply prudent.
At our scale, partnering with our vendors is a necessity. We often operate in areas that, while they are technically within specification, are toward the higher end of the range.
Re:Network Instability in NYC2 on July 29, 2014
by insaneirish on Sunday August 03, 2014 @12:38AM
(#8127043)
I urge you to seriously reconsider your design. VPC/MLAG/etc. are attractive because of the ease of making any VLAN available anywhere without the traditional spanning tree nightmare of doing so. However, you're trading one nightmare for another. Your core or aggregation switches are now stateful, and you've just experienced the results of stateful network components not failing cleanly.
I'm not saying an overlay (e.g. VMware NSX née Nicira) is the solution either. I'd suggest rethinking why you need any VLAN anywhere, which is likely the reason for VPC/MLAG. If I'm wrong and you don't put the same VLAN in multiple cabinets, then rip VPC/MLAG out immediately and switch to a pure L3 design. It is so much simpler and consistent with industry best practices.
Re:Network Instability in NYC2 on July 29, 2014
by jwatte on Saturday August 02, 2014 @11:03PM
(#8126888)
Which vendor? I have experience with two brands in similar condos, one good, one not so much...
Re:Network Instability in NYC2 on July 29, 2014
by imbriaco on Sunday August 03, 2014 @12:07AM
(#8126983)
I prefer not to publicly name the vendor. This is about problems on our network, and it's ultimately our responsibility.
Re:Network Instability in NYC2 on July 29, 2014
by akg_67 on Saturday August 02, 2014 @11:40PM
(#8126943)
Requesting Failure Analysis (FA) and Root Cause Analysis (RCA) from equipment provider is pretty standard after major incidents. It helps customer understand the issue better and put process in place to handle similar incidents. Similarly, vendors do it willingly because it allows them to learn more about issue and either address issue with product enhancement or put out alerts to their service org/ other customers with potential troubleshooting and resolution guidelines.
"We are working very closely with our networking partner to understand the nature of the failure, assess the chances of a repeat event, and to begin planning architectural changes for the future."
"Our initial focus was on verifying the configuration so we initiated a line-by-line configuration review by engineers at our network partner"
Yikes. You get what you pay for.