Linux isn't the only limitation on line rate traffic. Many gigabit Ethernet switches can't handle line rate min sized gigabit traffic. Mine is an otherwise good Asus switch that only does 1,000,000pps instead of 1,488,095 pps. I would bet similar restrictions could exist in Ethernet cards too.

Yep, I read that the Intel 82599 can/can't (I forget which) hit line rate for packets that are a multiple of 64 bytes. (Care to guess the DMA burst size?) These kind of quirks have been documented in almost every NIC.

He calculates 476mbps at the IP level on 1gbps ethernet, but continues…

Note that this disparity only happens because we are using very small packets, where per-packet overhead dominates. In normal Internet usage, packets are 500-bytes in size, and the per-packet overhead is usually ignored.

So, to one digit of precision, gigabit ethernet is still gigabit ethernet.

Packets are not 500 bytes in size, not even as an average (mean) or median. Many studies of frame length distribution show 64 bytes (for the Ethernet frame) as the most common length, probably because of TCP ACKs. Next most popular is 1518 bytes, the Ethernet maximum, probably because of bulk transfers over TCP.

Also, jumbo frames are technically not Ethernet. The IEEE revised the 802.3 spec to increase the maximum envelope from 1518 bytes up to around 2000 bytes to accommodate VLAN stacking and other encapsulating schemes. But jumbos -- usually 9000 or 9216 bytes -- aren't legally Ethernet.

Correct: the packet size distribution over any aggregated Internet link is typically bi-modal: a peak at around 64 bytes (acks) and one at around 1500 bytes (transfers that fill the MTU).

The article seems to me very approximate.

Specifically, it was extended to 1522 bytes in 802.3ac.

Anyone familiar with why Linux in particular can't hit the limit?

I believe it has to do with the penalty going between kernel and userspace because I remember repeatedly seeing novel projects that blow past these limits suggest pushing the whole networking stack to userspace. But I could be wrong.

Interesting question, hope somebody chimes in.

Could very well be hardware- or driver-dependent, and obviously the switch or device on the other side of the cable has a say in it too. There's an awful lot of pluggable stuff in there to support everything in the market. Really this thing needs to say "I only measured 1.3Mpps between two 3.11 kernels on IVB boxes with RTL8168 devices connected via a Netlink blah blah blah switch" to have any diagnostic meaning.

He addressed the kernel / userspace limit by noting that 10GigE card performs better. At this level, I'd start checking driver settings — particularly since, as he noted, this is rather different from the traffic most things are optimized for.

Novell projects or Novel projects?

The limitation you see on the number of pps is mostly due to software and hardware artifacts. With careful tuning (interrupt affinities, etc.), or bypassing a bunch of software layering (e.g. Netmap), or special stacks (e.g. Intel DPDK) you should be able to hit "line rate" packet processing.

No mention of MTU?

This is why there are jumbo frames for bandwidth (not latency) intense operations.

It looks like MTU isn't mentioned because the author is looking for the minimum possible packet for port scanning

I'm told modern gigabit ethernet devices derive practically no benefit from jumbo frames, and the pitfalls of using them are legion.

Each frame is a CPU-hogging interrupt so if your bottleneck is the CPU, then jumbo frames will help by moving more data for the same number of frames. This happens easily on routers with low-power CPU.

This is only true for very basic Ethernet hardware. Various interrupt mitigation schemes came along in the late 90s along with gigabit ethernet. Even low-end Realtek gigabit chips have various segmentation offload and interrupt coalescing features these days.

Those schemes have varying levels of effectiveness. I personally experienced this wasn't a given even as late as the 2005-2010 era, when our purchases were shifted towards certain models specifically due to the issues with drivers or buffer limits.

I learned never to underestimate how much functionality a major vendor will be willing to cut if it saves $.10 per unit shipped…

Nice write up, so you're saying that even though you can push the limits of Ethernet locally, it won't help you once you hit a router.

I would love to have access to a Gbps connection to the internet... stuck with my 15Mbps ADSL.

15Mbps? LUXURY.

Nice write-up. I wonder if there's any market in spec-breaking routers that can accommodate smaller inter-frame gaps, etc.? What use is there for this that is not covered by jumbo frames?

using jumbo frames kind of goes the opposite way the author has intended, right?? in their tests they are aiming for highest PPS - not the most efficient use of the link...

i would agree that an interesting test would be to see how much data can be transmitted in 1 minute with jumbo frames combined with the smaller gaps...

edit: oh and 15Mbps is decent enough, would love fibre to the home though.

Do you mean for internal use only? Wouldn't that be a moot point once it hits the next hop?

There's not necessarily one next hop. If you can stuff more packets than standard out to your upstream, and the upstream delivers the packets over multiple interfaces, it can pad the packets out to the standard length, and you get > 100% utilization.

I understand the original purpose of the minimum packet length (needed for reliable collision detection), but I don't think it's really necessary for gigE, where I believe full duplex, point to point connections are mandated. I also don't think the minimum size would be sufficient to guarantee collision detection in a hypothetical shared medium at the allowed bitrate and distances. Frame to frame delay on the other hand, seems like it would still be useful , although maybe it could be tuned.

Good answer +1, I was sort of thinking about this but I didnt have the knowledge.

I've no idea. That's why I'm asking. I guess internal use, yes, i.e.: for high-throughput local network infrastructure. I guess just going to 10GbE or fibre would be a cheaper method.

Some proprietary applications of the gigabit Ethernet PHY use shorter inter-packet times, but those aren't compatible with regular Ethernet switches or NICs.

The speed of light. (299,792,458 metres per second)