I imagine that once they've trained the floating-point models, they'll then quantize them into integers to make inference faster. It's not something I've done, but I imagine that the limited range of the of integers may cause problems (though they say in the paper that the 16-bit product can be accumulated to something that's 32-bit). The features to do this will be coming fairly soon to regular TensorFlow too.[1]

[1] https://youtu.be/0r9w3V923rk?list=PLOU2XLYxmsIKGc_NBoIhTn2Qh...