There aren't any (edit: consumer) GPUs with PCIe5 yet, though they probably aren't far off. Plenty already have PCIe4 though.
Some low-end or middle range GPUs really only use 8 lanes, because that's how the chip is designed. Fewer lanes -> less silicon area for the pcie logic needed -> cheaper.
The chips which use 16 lanes take advantage of it and can saturate the link.
Nvidia is withholding new releases but the current hardware has more legs with new matrix implementations. Like FlashAttention doing some significant improvement every 6 months.
Nvidia could make consumer chips with combined CPU-GPU. I guess they are too busy making money with the big cloud. Maybe somebody will pick up. Apple is already doing something like it even on laptops.
https://www.datacenterknowledge.com/data-center-hardware/nvi...
"PCI Express 4.0 x8"
[root@gh200 nvbandwidth]# nvidia-smi | grep GH200
| 0 NVIDIA GH200 480GB On | 00000009:01:00.0 Off | 0 |
[root@gh200 nvbandwidth]# ./nvbandwidth | grep -E 'host_to_device_memcpy_sm|device_to_host_memcpy_sm' | grep ^SUM
SUM host_to_device_memcpy_sm 357.45
SUM device_to_host_memcpy_sm 352.05
[root@gh200 nvbandwidth]#
d2d is much higher:
[root@gh200 cuda-samples]# ./bin/sbsa/linux/release/bandwidthTest --dtod | grep -B1 32000
Transfer Size (Bytes) Bandwidth(GB/s)
32000000 2294.8
[root@gh200 cuda-samples]#
PCIe 3.0? What?
https://cowfreedom.de/#appendix/computer_specs/
> GeForce GTX 1050 Ti with Max-Q Design (PCIe 3.0 x16) (2016)
> Intel Core i5-8300H (2020)
This is a low-price 8 year old GPU and a 4 year old CPU. And he seems to be including loading the data to GPU. Newer cards have wide PCIe 5.0 or some faster interconnect, like Nvidia Grace-Hopper.
Also he is comparing his own CUDA implementation. He should use one of the many available in CUBLAS/CUTLASS. Making a good CUDA GEMM is a very difficult art and very hardware specific