The thing I've seen is very limited (NVIDIA GPUs offer up to 8x more half precision arithmetic throughput when compared to single-precision, thus speeding up math-limited layers.[1]) which is probably true.
The problem with performance improvements is the diminishing returns part of Amdahl Law: an 8x improvement in math performance will just mean the math part becomes less important in terms of absolute performance.
In any case, I've found NVidia's claims in the machine learning area to be pretty good. Like most claims you have to read carefully to see exactly what the claim is, but that's not uncommon with performance claims.
[1] https://docs.nvidia.com/deeplearning/performance/mixed-preci...
[2] https://en.wikipedia.org/wiki/Amdahl%27s_law#Relation_to_the...
Right but I’ve benchmarked the best case scenario, ie a large GEMM call in C++, and still not seen anywhere close to 8x. I’ve never seen a code example, no matter how limited, showing a 8x speed up.
See cuBLAS mixed-precision GEMM.
On https://images.nvidia.com/content/technologies/volta/pdf/vol... they claim 1,525 images/sec (!)
Dell hit 5,243 images/sec with one of their 4 V100s servers, which comes to 1,310 images/sec per V100. I find it very believable that NVidia would get ~200 images/sec more, since Dell jumped 50% with a change in their CPU/GPU connection topology.
See https://www.dell.com/support/article/en-au/sln317397/deep-le...