Mixed precision training can give you tensor core speedups. Paper: https://arxiv.org/abs/1710.03740 Toolkit which implements it on top of Tensorflow: https://github.com/NVIDIA/OpenSeq2Seq