Of course, the underlying TCP latency is significantly lower. Using Microsoft's "Latte.exe" testing tool, I saw ~50 μs in Azure with "Accelerated Networking" enabled. As far as I know, they use Mellanox adapters.
Something I found curious is that no matter what I did, the local latency wouldn't go below about 125 μs. Neither shared memory nor named pipes had any benefit. This is on a 4 GHz computer, so in practice this is the "ultimate latency limit" for SQL Server, unless Intel and AMD start up the megahertz war again...
It would be an interesting exercise comparing the various database engines to see what their latency overheads are, and what their response time is to trivial queries such as selecting a single row given a key.
Unfortunately, due to the DeWitt clauses in EULAs, this would be risky to publish...
Modern processors can do ~2.1GB/s per core for AES-256-GCM (about 2x what they can do for AES-CBC).
Both EPYC and Xeons have 2-AES units per core now. But CBC can only effectively use one of them at a time. (Block(n+1) cannot be computed until block(n) is done computing. Because Block(n) is used as input into Block(n+1) in CBC mode).
AES-256-GCM can compute block(n) and block(n+1) simultaneously. So you need to use such a parallel algorithm if you actually want to use the 2x AES pipelines on EPYC or Xeon.
Very often, you'll see n-tier applications where for some reason (typically load-balancers), the requests are muxed into a single TCP stream. In the past, this improved efficiency by eliminating the per-connection overhead.
Now, with high core counts and high bandwidths, some parallelism is absolutely required to even begin to approach the performance ceiling. If the application is naturally single-threaded, such as some ETL jobs, these single-stream bottlenecks are very difficult to overcome.
In the field, I very often see 8-core VMs with a "suspicious" utilisation graph flatlining at 12.5% because of issues like this. It boils my blood when people say that this is perfectly fine, because clearly the server has "adequate capacity". In reality, there's a problem, and the server is at 100% of its capacity an the other 7 cores are just heating the data centre air.
I recently did a bunch of tests to see what the "ultimate bottlenecks" are for basic web applications. Think latency to the database and AES256 throughput.
Some rough numbers:
- Local latency to SQL Server from ASP.NET is about 150 μs, or about 6000 synchronous queries per second, max.
- Even with SR-IOV and Mellanox adapters, that rises to 250 μs if a physical network hop is involved.
- Typical networks have a latency floor around 500-600 μs, and it's not uncommon to see 1.3 ms VM-to-VM. Now we're down to 800 queries per second!
- Similarly, older CPUs struggle to exceed 250 MB/s/core (2 Gbps) for AES256, which is the fundamental limit to HTTPS throughput for a single client.
- Newer CPUs, e.g.: AMD EPYC or any recent Intel Xeon can do about 1 GB/s/core, but I haven't seen any CPUs that significantly exceed that. That's not even 10 Gbps. If you have a high-spec cloud VM with 40 or 50 Gbps NICs, there is no way a single HTTPS stream can saturate that link. You have to parallelise somehow to get the full throughout (or drop encryption.)
- HTTPS accelerators such as F5 BIG IP or Citrix ADC (NetScaler) are actually HTTPS decelerators for individual users, because even hardware models with SSL offload cards can't keep up with the 1 GB/s from a modern CPU. Their SSL cards are designed for improving the aggregate bandwidth of hundreds of simultaneous streams, and don't do well at all for a single stream, or even a couple of concurrent streams. This matters when "end-to-end encryption" is mandated, because back end connections are often pooled. So you end up with N users being muxed onto just one back-end connection which then becomes the bottleneck.