News for hackers. Stuff that matters.
Re:Jax – Composable transformations of Python and NumPy prog... by tbenst on Wednesday April 08, 2020 @11:38AM (#22813878)
Has anyone benchmarked Jax? Curious how it compares to PyTorch for nontrivial networks, say ResNet.
Re:Jax – Composable transformations of Python and NumPy prog... by gdahl on Wednesday April 08, 2020 @03:01PM (#22815893)
Check out the Flax ResNet50 example: https://github.com/google/flax/tree/master/examples/imagenet

It runs about as fast as any of the other popular machine learning frameworks, occasionally faster.

Disclaimer: I work for Google and use JAX, although I'm not on the Jax team.

Re:Jax – Composable transformations of Python and NumPy prog... by komuher on Wednesday April 08, 2020 @11:54AM (#22814037)
It is compiled to XLA so should be a lot faster then pure PyTorch but probably will be slower then TVM (https://tvm.apache.org/) i can prepare some benchmarks in next few days if u are interested :)
Re:Jax – Composable transformations of Python and NumPy prog... by tbenst on Wednesday April 08, 2020 @12:17PM (#22814270)
That would be wonderful if you’re able to! Also doubles as a good intro to Jax :). Please feel free to tweet at me (@tbenst) or email [same username at stanford dot edu] if you do get around to it.
Re:Jax – Composable transformations of Python and NumPy prog... by komuher on Wednesday April 08, 2020 @12:38PM (#22814514)
Sure (dont have twitter yet) but will post it here on hacker news in next week probably :)
Re:Jax – Composable transformations of Python and NumPy prog... by m0zg on Wednesday April 08, 2020 @05:49PM (#22817566)
> XLA so should be a lot faster

I've yet to see anything get "a lot faster" because of XLA. It's a ton of complicated code, but then you end up spending the vast majority of time in NVIDIA's cuDNN anyway, so any benefits you might have hoped for will be marginal at best.

Re:Jax – Composable transformations of Python and NumPy prog... by p1esk on Wednesday April 08, 2020 @10:44PM (#22819324)
Re:Jax – Composable transformations of Python and NumPy prog... by m0zg on Thursday April 09, 2020 @01:26AM (#22820048)
Easily outperformed by the more traditional TensorRT, which TF also supports: https://devblogs.nvidia.com/tensorrt-integration-speeds-tens....

In fact, also seems to be outperformed by plain PyTorch using a single V100: https://github.com/NVIDIA/DeepLearningExamples/tree/master/P...

Re:Jax – Composable transformations of Python and NumPy prog... by p1esk on Thursday April 09, 2020 @07:26PM (#22828308)
Interesting. I wonder why there's such a difference between Nvidia Pytorch benchmark and Exxact results: Nvidia is more than twice faster for single GPU. V100 should only be ~10% faster than Quadro 8000. Either Exxact is incompetent, or Nvidia has some special sauce.
Re:Jax – Composable transformations of Python and NumPy prog... by m0zg on Thursday April 09, 2020 @08:27PM (#22828700)
FWIW, NVIDIA TensorRT pre-profiles the models before it runs them. I don't know how it does that exactly (that part is closed source) but I'd guess they just try different algorithms on each op individually (i.e. plain conv vs Winograd) and pick a good balance of speed and memory usage according to heuristics. On some nets this can make all the difference in the world, and ResNet50 is basically the most studied architecture in existence, so you can bet it's in every single benchmark for this kind of thing, and as such it receives disproportionate attention.
Re:Jax – Composable transformations of Python and NumPy prog... by p1esk on Thursday April 09, 2020 @11:22PM (#22829704)
I thought all frameworks can do this type of profiling (e.g. torch.backends.cudnn.benchmark = True).

Nvidia might have eliminated any potential data pipeline bottlenecks (with careful DALI tuning), but I'd still expect a lot less speedup. Maybe they compiled pytorch with certain tricks, and used newer CUDA/CuDNN code, idk.

Re:Jax – Composable transformations of Python and NumPy prog... by m0zg on Friday April 10, 2020 @01:06AM (#22830151)
TRT profiling is more extensive. On the model I'm currently working with (which runs on Jetson Xavier), initial TRT profiling takes something like 4 minutes. The model is an object detector. You can save the result, but it's hardware dependent then, so the resulting model is only optimal for the particular hardware it was optimized for. I cache it on disk - I can't wait 4 minutes every time I run a test.

PyTorch, as far as I can tell, does much lighter cuDNN profiling. It's more pareto optimal, I suppose, but the benefit is nowhere near as significant.

Another framework which does amazing optimization (but on the CPU) is OpenVINO. Normally I don't expect much on the software side from Intel, but this thing really blows the doors off everything else if you don't have a GPU at your disposal, provided that you have an Intel processor. The wayt they do it is they generate kernels that fit your data, but not the way XLA does it. They hand code them in a DSL that produces assembly, using Xbyak, and incorporate their deep knowledge of Intel hardware into that. When it's time to run the model, that DSL spits out optimal kernels just for that particular model. It's pretty neat work, IMO.

Re:Jax – Composable transformations of Python and NumPy prog... by p1esk on Friday April 10, 2020 @04:34PM (#22837076)
I see, thanks. In my day job I develop hardware accurate simulations of a deep learning accelerator. This involves looking at a Spice model, simplifying it into some set of abstractions using Numpy, then accelerating this Numpy code using GPUs. Currently I'm porting a resnet-50 model from Numpy to Pytorch, and the next step is to speed up the Pytorch code (because right now I get ~1 image per second, which is about 10 times better than Numpy). Perhaps I should look into porting the model from Pytorch to TensorRT.
Re:Jax – Composable transformations of Python and NumPy prog... by m0zg on Friday April 10, 2020 @09:09PM (#22838961)
If you're working with pytorch, porting basically means export to ONNX. Sometimes you'll run into an op that doesn't work with ONNX, but there are a lot fewer of those in TRT7. Unfortunately I have to work with TRT6, so I have to use PyTorch 1.2 and be "creative" to work around TRT6 bugs. That said, it could very well be painless for you. No reason not to try. Just export the model, and benchmark it with `trtexec`, in both fp32 and fp16. An hour of work at most.
Re:Jax – Composable transformations of Python and NumPy prog... by p1esk on Wednesday April 08, 2020 @02:48PM (#22815769)
I'm very interested in seeing Resnet results. Especially FP16 precision running on tensor cores (V100 cards). Please use synthetic input vectors so that the input pipeline is not a bottleneck (as it is often the case, and it varies per framework).