Do you know why TF is getting poor GPU util? Is it the pipeline feeding data to the core compute ops, inefficiencies in the core compute ops, or something else?
Just wanted to chime in on TensorRT, it's a well supported product and it's different than gpu-rest-engine. This GitHub repo is simply an example of how to use TensorRT in a specific situation.
Question about TensorRT: It takes in Caffe trained models. Is this BVLC Caffe or NV-Caffe? In my experience the models aren't compatible between the two.
Re:Google TPU Performance Analysis
by alexnewman on Wednesday August 23, 2017 @01:13PM
(#15083263)
Either but I've never tried
- TF can't do anything quantized on GPUs. It just switches back to to the CPU/TPU. - TF gets relatively poor utilization of the GPU and tends to not be careful with memory use. - I was able to do certain types of classification hundreds of times faster by seeing what TF was doing it and hand writing it in OCL. Using https://docs.rs/ocl/0.14.1/ocl/. It's a super cool library for rust. Also users should checkout tensorRT https://github.com/NVIDIA/gpu-rest-engine/tree/master/tensor.... It's not super well supported and may go away, but it is fast