At around 4 GHz and 2 IPC, it's about 1000 CPU instructions per I/O.

Assuming you have an hardware DMA ring buffer for I/O that is directly mapped in userspace, the only thing that is really needed is to write the operation type, size, disk position and memory position to the ring buffer, update the buffer position and check for flush, doable in around 8 CISC instructions (plus the slowpath), so around 100x inefficient.

Without the direct mapped ring buffer and with a filesystem, you need a kernel to translate from uring to the hardware ring buffer, and here it still seems around 10x inefficient as around 100 instructions should be enough to do the translation (assuming pages already mapped in the IOMMU, that you have the file block map in cache, and that the whole system is architected to maximize the efficiency of this operation).

> Assuming you have an hardware DMA ring buffer for I/O that is directly mapped in userspace, the only thing that is really needed is to write the operation type, size, disk position and memory position to the ring buffer, update the buffer position and check for flush, doable in around 8 CISC instructions (plus the slowpath), so around 100x inefficient.

This type of thing is not measured in instructions anymore, it's measured in off-board operations and latencies. Cache misses, latencies to poke MMIO and get back interrupts if necessary, DMA transfer to complete, device access time. In this case it seems the hardware is theoretically capable of about 12M so the core mostly be just waiting for that.

Will DMA from external devices pass L3 or some other CPU cache? Or are all accesses to the freshy DMAed-to memory be cold

It's worth noting that there's some batching involved, so it's not 125ns for processing one io, then again 125ns for the next. Both combining some of the work, and superscalar execution is necessary to reach this high numbers...

To expand on the batching details, I haven't seen exactly what he's doing here, but historically the numbers quoted are from benchmarks that submit something like 128 total queue depth in alternating batches of 64. The disks hit max performance with a much lower queue depth than 64. So it's very unrealistic perfect batching that you'll never see in a real application. The workload is only a useful tool for optimizing the code, which is exactly what he's using it for.

The Intel Optane drives he's testing with hit max performance at pretty low queue depths, but flash-based SSDs with similar throughput will require those high queue depths.

I think you need batching for the nvme doorbell alone at anything close to these kinds of rates...

For some workloads it's not that hard to have that deep queues. What's harder is to know when to use them and when not. There's really not enough information available to make any of this self-tuning.

Faster than DRAM from the 90's.

The access time has actually not changed a lot, it's just that transferring the smallest read a contemporary processor would issue - one cache line (~64 bytes) - at somewhere between 500-1000 MB/s takes around 100 ns already. That transfer-related latency has been reduced drastically.

Didn't /RAS timing drop from like 120ns to 10ns too? That's what I was thinking about. Then DDR, and that bus size is also 16 times wider.

Wasn't tRAS always around 2-3x tCL?

Modern SSDs generally have more bandwidth than DDR2.

I'd hope so lol

IO is 512B sectors here? Or 4KiB?

EDIT: 512, looking at the screenshot..