I've yet to see anything get "a lot faster" because of XLA. It's a ton of complicated code, but then you end up spending the vast majority of time in NVIDIA's cuDNN anyway, so any benefits you might have hoped for will be marginal at best.
Almost double speedup for FP16 Resnet-50.
In fact, also seems to be outperformed by plain PyTorch using a single V100: https://github.com/NVIDIA/DeepLearningExamples/tree/master/P...
Nvidia might have eliminated any potential data pipeline bottlenecks (with careful DALI tuning), but I'd still expect a lot less speedup. Maybe they compiled pytorch with certain tricks, and used newer CUDA/CuDNN code, idk.
PyTorch, as far as I can tell, does much lighter cuDNN profiling. It's more pareto optimal, I suppose, but the benefit is nowhere near as significant.
Another framework which does amazing optimization (but on the CPU) is OpenVINO. Normally I don't expect much on the software side from Intel, but this thing really blows the doors off everything else if you don't have a GPU at your disposal, provided that you have an Intel processor. The wayt they do it is they generate kernels that fit your data, but not the way XLA does it. They hand code them in a DSL that produces assembly, using Xbyak, and incorporate their deep knowledge of Intel hardware into that. When it's time to run the model, that DSL spits out optimal kernels just for that particular model. It's pretty neat work, IMO.