Interestingly, the ENA driver has #defines for speeds up to 400 Gbps.
My guess as to why EC2 instances are limited to 25 Gbps is that it's a matter of balancing overprovisioning and the need to avoid having a single instance eat too much of a rack's bandwidth. I don't know how much bandwidth they have going to each rack, but there's a limit to how much it makes sense to provision; if typical bandwidth is on the order of 10 Gbps per rack (say, 80 instances pushing 125 Mbps on average) then you might want to provision 200 Gbps/rack and limit each instance to 25 Gbps rather than provisioning 1 Tbps/rack and limiting each instance to 100 Gbps.
(Numbers above are completely invented; I don't have any internal knowledge of how Amazon's networks or datacenters are set up.)
Also, are you talking about Annapurna in it's pre-acquisition form or new one? AWS talks about new custom asics and multiple ARM SoCs on their Nitro system.
AWS aren’t alone in this, and actually do pretty darn well compared to their competition - we had a nightmarish time a few years back with exactly this with a VPS provider - half of every second the traffic to the memcached cluster would just stop. Turned out they’d set hard limits on packets/sec to avoid oversaturating the host, so the advertised Gbps interconnect was actually 50Mbps when you saturated the packet scheduler.
Do GCP, Azure, or any other cloud providers offer 100G networking?
AWS Nitro allows 5G/bit per flow. And then maxes out at 25G/bit. I know GCP does something similar.
Also, pretty sure that is false regarding Azure, they have a small availability of Infiniband, but, that is not on their general compute platform and has a narrow use case/many restrictions. Azure has had the worst networking performance from my experience and only had 10GbE NICs (it's been a while though)
"Azure is breaking the speed barrier in cloud connectivity. ExpressRoute Direct provides 100G connectivity for customers with extreme bandwidth needs. This is 10x faster than other clouds."
https://azure.microsoft.com/en-us/blog/azure-networking-fall...
Can sb confirm that? Have a useful case in mind.
Here is a demonstration of a software stack that can scale toward hardware limits without relying on a particular vendor https://www.slideshare.net/SeanChittenden/freebsd-vpc-introd.... This approaches 100G line rate for large packets which is what it was optimized for. I don't know PPS at low packet size but do know what would be required to optimize that use case and it could be done pretty quickly.