There is also some work on pure binary networks, xnornet and binarynet. Low precision doesn't seem to effect these networks as long as they are trained with binary weights in the first place.
A vendor-made benchmark where the vendor is outperforming other vendors.
Let's not jump to conclusion on what's really fastest.
The reason you don't typically have FP16 on CPUs is that in the CPU world, cache dominates die area and power usage. The floating point ALU component of a CPU is relatively tiny. Still, even in the CPU world, Intel recently added instructions to convert to/from FP16 before operating on it, because they know there is a memory bandwidth advantage in using a smaller floating-point format.
https://dennisforbes.ca/index.php/2017/04/11/floating-point-...
Most desktop chips offer little advantage for FP16 (at best that the source memory footprint is smaller, maybe better cache hit rates), but newer GPUs can actually blaze. If it has the magnitude/precision needs for a task, it's a big win.
(In fact there are even some experiments with a very low number of bits - can't find the link now though)
I'm interested more in understanding why Caffe2 would outperform Theano, Tensorflow, MXNet, etc. once Volta chipsets are generally available, beyond early pre-release optimization, particularly when most of the front-runners are already leveraging / taking into account NCCL, CuDNN, NVLink, etc. When the burden of adding support for new NVIDIA primitives is so low, what gives Caffe2 an advantage beyond an ephemeral "we were partners with NVIDIA first" one-up that would last for a couple of months at most?
(Apologies in advance if this post sounds overly negative, but I am constantly evaluating the current crop of frameworks for the trade-offs they enforce on the problem space, and a definitive answer would be very helpful.)