I fixed the installation and ran a benchmark again: https://news.ycombinator.com/item?id=21666411
The benchmark is not CPU bound, as evidenced by looking at rocm-smi, nvidia-smi (GPU usage at 100%) and htop (low CPU usage).
Nice. Your results could be more reproducible, if you'd include CuDNN version. Turing architecture support (and optimizations specific to DNN training) are still relatively recent and there are differences in performance between the versions [1]. Multiplying matrices is kinda tricky ;) [2]
[1] https://developer.nvidia.com/cudnn . [2] https://scholar.google.com/scholar?as_ylo=2019&q=nvidia+gemm
It is just that "more normal" result that you would expect to get, would be a drastic differences in performance. Note, Cvikli comment on "May 12" on that GitHub page:
* The result with RNN networks on 1 Radeon VII and 1080ti was close to the same
* Comparing convolutional performance the 4AMD and 4Nvidia, difference got really huge because of cuDNN for Nvidia cards. We can get more than 10x performance from the 1080Ti than the Radeon VII card. We find this difference in speed a little too big at image recognition cuDNN, I can't believe that this should happen and the hardware shouldn't be able to achieve the same.
Note also in that comment, it looks like 4xGPU system just didn't work.
It is hard to fully utilize GPU resources. Getting same result normally means that the same bottleneck is being hit. Algorithmic differences could result in drastic performance differences (10x). And as a final point of small things making a difference, note it is 1080Ti in that comment.