Re:Bfloat16 – Hardware Numerics Definition [pdf]
by twtw on Saturday November 17, 2018 @01:59PM
(#18476837)
I would be extremely surprised if the motivation for putting bfloat16 in tensorflow was not the TPU. That first public commit was ~1.5 years before TPUv2 was announced at I/O, so it was almost certainly already in development.
[ Parent
| on HN ]
Re:Bfloat16 – Hardware Numerics Definition [pdf]
by vrv on Saturday November 17, 2018 @03:02PM
(#18477087)
bfloat16 was first in DistBelief, so it actually predates TensorFlow and TPUs (I worked on both systems). IIRC the motivation was more about minimizing parameter exchange bandwidth for large-scale CPU clusters rather than minimizing memory bandwidth within accelerators, but the idea generalized.