francoisLabonte

From my professional experience of programming and debugging networking equipment, this could be a switch/router with a buffer with bad memory (stuck bit maybe). The better chips have CRC/Parity/ECC to cover such issues but there are always those magical choke points where the past CRC is tossed and the new one is generated that can leave a gaping hole. The tricky part is how often is this bad memory buffer used...

I would use traceroute to find a common bad point for everyone. It is also possible that the networking point where the problem occurs is invisible to traceroute as it could be part of a provider network probably MPLS but at least the common ends of the tunnel would be visible.

The fact that it is a a specific interval indicates a stuck bit in memory.

Some good previous public stories about such incidents https://www.verizondigitalmedia.com/blog/being-good-stewards... https://twitter.com/cperciva/status/1309568337408454658

After more consideration it is much more likely to be LPDDR4X just like the recently released A14 chip. It would seem unlikely that they would have developed a brand new memory interface.

My guess is that they are using up to 2 HBM2 memory stacks (from the picture). Each is limited to 8GB . If they were to go to HBM2e in M2 they could get up to 2x24GB. The biggest advantage of HBM is lower power per bit as the signals are all on the package running at a lower frequency.

The memory market is getting fragmented with Nvidia having seemed to moved from HBM2 to GDDR6X (Micron being only supplier). LPDDR super low power (cell phones) and DDR4/5 for rest of market...

Flex Ethernet is that attempt to standardize the lane stripping at whatever speed. Unfortunately that effort has grown to incorporate a kitchen sink worth of other features like groups and shaping... Which raises the hardware implementation cost and lowers the adoption rate.

https://www.oiforum.com/wp-content/uploads/2019/01/OIF-FLEXE...

At the 100G/200G/400G/800G specs are highly repetitive and the only points of debate are the type of FEC required given the high speed signals, signal loss for cost effective board/cable/optics and how parallel the FEC will be (tradeoff of hardware implementation cost and added latency). FEC has been 2 engines in parallel since 400G.

The pirates are a spoof of another french Belgian comic named Redbeard (Barbe Rouge) https://en.wikipedia.org/wiki/Redbeard_(comics) The wikipedia page even has the Asterix parodies next to the actual ones.

Not that it excuses anything.

As one of the authors of the original paper the data is based on I have fond memories as a graduate student of standardizing the performance data to SpecInt which did not exist till the 2000s. So for the older processors that were no longer available I looked hard for processors that had both SpecInt and older benchmarks like Dhrystone and MIPS per seconds to normalize them to SpecInt performance.

The same data was used by other students of my advisor to create the Cpu database which has some more data like cache sizes and FO4.

http://cpudb.stanford.edu/

If you want to build a non-blocking Clos network of 2N ports with switches with N ports you need to take 4 switches in the front with N/2 ports and 2 switches in the back interconnecting all the ports to the back for a total of 6 switches, so not even including cables a modular switch providing 2N ports is worth 6x per port than a fixed switch... The number are a little better for bigger but approach at a cost per port of 3x per port not including cables. In my experience modular switches are more in the ball park of 2.5-3x per port. I am not including other possible benefits that modular switches can have non-blocking fabrics.

Tons of problems with costs...

1) Copper cables only go to 7m because attenuation is too high after 7m to recover signals at 10G/25G lanes, get ready for lower length with 50G PAM4 2) Transceivers are free... no they are not fiber is one aspect of the cost but transceiver dominate 3) Missing Active Optical Cable (they are made of transceivers that do not quite meet the spec and made into a 'cable' with a fiber. These are now very important in saving costs

What is really missing here is that you can build modular switches out of multiple chips that are a lot more cost effective than connecting single chip boxes mostly because the internal links in a modular chassis in connectors are a lot more cost effective than any cable or optics. This makes the Clos topology with Top or Rack using copper cables and spines out of modular switches with up to 576x100G ports in today's technology winners every time for datacenters. The supercomputing world is still stuck in requirements of extreme low latency and hence hitched to Infiniband or other specialty networks with devices with low number of ports.

The real debate is between OSPF and QSFP-DD as they are the only ones with a shot at high density.

QSFP-DD is backwards compatible so that you can run your 400G port as a 100G or 40G with 4 lanes at 25G/10G. OSFP will be able to do the same thing but with an adapter from OSFP to QSFP

But what this article fails to mention is that OSFP is the connector of the future that will support 8 lanes of 100G to support 800G ethernet with good signal integrity and capability to support high power optics for longer distances. ( This looks to be in 2020 )

So QSFP-DD is a 1 generation connector...

OSFP will see us through 400G and 800G.

Disclaimer I work at Arista Networks who is bullish on OSFP.

For more info see Andy Bechtolsheim's talk on 400G optics at OCP

https://www.youtube.com/watch?v=Kotu6B7AQpk

Hopefully you guys have ECC on your 64MB of SRAM, otherwise the meant time to bit flip due to Single Event Upset (SEU) is around 400 days ( based on 200 Fit/Mb/Billion Hours from previous experience ).

Like you said the host might have small buffers and without Pause it would drop, but who's supposed to buffer the packets, the cheap switch with 16kB of buffers and super idiotic buffer configuration such that everyone else on that switch gets paused?

You seem to think that it's bad to drop packets in the nic and while some nic might have buffers that are too small but in general you should drop. If you use TCP the window will adjust to whatever your bad nic and embedded system can handle. At least you won't affect the others by spreading pause like a cancer ( can you tell I am cynical on pause )

Usually on a switch you can usually drop packets based on the number of packets destined to a port and packets buffered per input port. This is how you can avoid head of line blocking but again if you are right with 16kB that's barely enough for a jumbo packet (~9200B)... geez that's depressing.

Some datacenters enable Priority Flow Control (PFC) which is different in that it pauses only the traffic with a specific PCP ( Priority in 802.1Q vlan tag ). They assign storage traffic a specific vlan priority and treat it as lossless with flow control but the rest of the traffic is unaffected.

The mechanism here Pause is an abomination which should never be enabled.

Ah... The problems of crappy consumer ethernet equipment ( I work at an ethernet switch vendor so excuse the rant )

What is likely happening is that your switch is configured by default to implement both rx and tx pause. What is happening is that your TV who's also erroneously ( in my opinion ) configured to transmit pause goes bonkers, starts sending pause to your switch. Your switch then starts buffering packets for your tv until the buffers are full and then starts transmitting pause to everyone else including ports. The switch must have some horrible buffering policy where one port ( the tv port ) can hog all the buffers and deprive every other ports of being able to send...

Now the kicker is that the way every endstation implements pause is this... Notice the pause quanta in the Pause packet is in units of 512 bits and in the packet you captured it is set to the maximal value of 65535 which is on a 100Mbps port ( presuming 100Mbps since the Mediatek has 4x100Mbps (Fast Ethernet) and 2x1Gbps ( Gigabit Ethernet ) that computes to >>> 512*65535/100.e6 = 0.3355392seconds

A normal Pause sends this packet periodically and once it has buffers to receive will send a pause with a quanta of 0 meaning cancel previous timer... but if it's malfunctioning who knows if it ever will...

The sad part is that I don't even know what to recommend for a good consumer level switch that has good defaults or configurable defaults and sane buffer config... Mine is a dinky one probably vulnerable to this problem as well... Need to do some research.

What nobody talks about is the very good reason why employers give only 90 days after leaving a company to exercise options is that there is also a tax liability to the company for an employee exercising an option. Usually employer has to pay employment tax, now if you have a lot of options still unexercised from former employees and your stock has appreciated a lot the company can be on the hook for a lot of taxes. The company prefers only being on the hook for current employees.

The true solution is to give stock options that can be exercised early as long as the value of the stock is very low such that an employee's hiring bonus after tax could cover the cost. There is no tax owed by the employee since he purchased shares with no gain and then you vest outright stock. Once the stock value goes up it would be best to grant RSUs of convertible notes that convert into stock.

In the networking world Fulcrum built some very low latency switch chips used in switch routers using asynchronous logic. The Alta switch chip was the last of that generation.

http://www.hotchips.org/wp-content/uploads/hc_archives/hc23/...

Intel acquired Fulcrum and has not had a new product. One can speculate that they were acquired in part for their experience and tool to design asynchronous pipelines.

In the DSP world Octasic makes DSPs that use asynchronous desisns:

http://www.octasic.com/technology/opus-dsp-architecture

Also note Spotify published their own stuff also on Arista hardware

https://labs.spotify.com/2016/01/26/sdn-internet-router-part...

Podcast where David Barroso talks about it:

http://blog.ipspace.net/2015/01/sdn-router-spotify-on-softwa...

Arista blog post:

https://eos.arista.com/spotifys-sdn-internet-router/

Disclaimer I work at Arista