briggers

Machine learning productionization: https://paulbridger.com

Absolutely yes.

I write fairly deep ML performance tuning articles at https://paulbridger.com and the (many) hours I've spent on each article have been hugely worth it.

Many people reach out to me via this work, and when we talk they already see me as an expert or already want to work with me.

I need to blog more, thanks for the reminder.

Before you start optimizing runtime performance: measure, trace, inspect, or whatever is appropriate to understand current performance.

Perhaps you are looking for something like this?

docker run -d -p 5000:5000 --name registry registry:2

https://docs.docker.com/registry/#:~:text=The%20Registry%20i....

I used to love doing this with Clojure, it’s an awesome way to increase productivity by a good chunk.

It’s less about saving the time to re-run something, and more about removing conceptual overhead (I think).

Yes! I do this and love it. Works locally or via SSH with no difference at all.

Nice one! I've long been interested in the ONNX serving path.

Yeah. A 2080Ti doesn't fit in your pocket or in your AR glasses but the same techniques and tools scale down.

BTW, this is pumping the same video file through the network - not just a single file. I don't measure latency, but this is not a deep pipeline so it's easy to calculate.

Mobile phones definitely since these days most of them have pretty powerful GPUs.

Great question, now I wish I'd recorded power consumption for all these experiments. Judging from cumulative hours of watching the output of nvidia-smi I've definitely seen a linearish relationship between utilization and power draw (with a non-zero floor of 30-40W).

Very practical question :) Exactly as you say, multi-stream throughput. Also for faster than realtime offline processing of video. Check the caveats section at the end of the post - DeepStream is probably not well suited to high throughput single-stream inference.

I looked into the GIL saturation as measured with gil_load (https://github.com/chrisjbillington/gil_load), but perhaps I should have included more numbers here.

To me, seeing the GIL held for 40% of time and significant time spent waiting on GIL by other threads was a fairly strong indicator. Keen to hear your thoughts/experience on it.

Great point - dependencies between frames are inherently problematic for many of these techniques.

Everything lostdog says. I've had experience speeding up tracking immensely using the same big hammer I talk about in the article - moving the larger parts of tracking compute to GPU.

Also, in a tracking pipeline you'll generally have the big compute on pixels done up front. Object detection and ReID take the bulk of the compute and can be easily batched and run in parallel. The results (metadata) can then be fed into a more serial process (but still doing the N<->N ReID comparisons on GPU).

Author here. As other commenters are saying, the Pytorch JIT and torchscript might be your friend here.

Alternatively, there are some quite fast OSS libraries for object detection. Nvidia's retinanet will export to a TensorRT engine which can be used with DeepStream.

Author here. I really appreciate your feedback.

Completely agree that almost none of the SoTA github repos are really ready for production and making this stuff work can be pretty hard.

Getting this done on C++ and moving up to the next level of performance is the focus of my next article :)

Nice one.

More useful to me would be something similar that operates on tensors on the GPU.

Doing image annotations on host/CPU often becomes a bottleneck.

It’s important to note this is just within visual range dogfighting/BFM. Also just guns, and also perfect enemy state information. Important progress but it’s super early. Beyond visual range/BVR, coordination with wingmen and many other higher-level tasks still to come (or are still secret).