Re:Go-msquic: A new QUIC/HTTP3 library for Go
by vitus on Wednesday February 19, 2025 @07:50PM
(#43109794)
I am not saying that we are wasting 25% of our compute costs on QUIC, but rather that we hit many other bottlenecks far before we hit that point.
> Having done the majority of a QUIC implementation myself, achieving (on the non-encryption portion) 10 Gbps (1.5x faster) seems trivial
Encryption is a huge part of the cost, especially with the general lack of NIC offload as I mentioned. Since the packet number changes across retransmissions, lossy environments exacerbate the problem since you need to re-encrypt retransmitted packets (which is not a problem you have in TCP). If you're not doing any sender-side pacing, then you're likely to overwhelm your NIC with microbursts.
100 Gbps per core is quite frankly unrealistic; the only way you get that even in TCP is if you have your smart NIC talking directly to your SSDs and bypassing main memory and the kernel TCP stack entirely, and all of your encryption is done by the NIC. But as mentioned, we don't have that level of NIC support.
Even 30 Gbps per core doesn't leave you much wiggle room if you have to perform AES via your CPU.
> Having done the majority of a QUIC implementation myself, achieving (on the non-encryption portion) 10 Gbps (1.5x faster) seems trivial
Encryption is a huge part of the cost, especially with the general lack of NIC offload as I mentioned. Since the packet number changes across retransmissions, lossy environments exacerbate the problem since you need to re-encrypt retransmitted packets (which is not a problem you have in TCP). If you're not doing any sender-side pacing, then you're likely to overwhelm your NIC with microbursts.
100 Gbps per core is quite frankly unrealistic; the only way you get that even in TCP is if you have your smart NIC talking directly to your SSDs and bypassing main memory and the kernel TCP stack entirely, and all of your encryption is done by the NIC. But as mentioned, we don't have that level of NIC support.
Even 30 Gbps per core doesn't leave you much wiggle room if you have to perform AES via your CPU.