Shameless plug, check out DOODS: https://github.com/snowzach/doods It's a simple REST/gRPC API for doing object detection with Tensorflow or Tensorflow Lite. It will run on a Raspberry Pi. It actually did support the EdgeTPU hardware accelerator to make the Pi pretty quick for certain models. They broke something so I need to fix EdgeTPU support but it's still usable on the Pi withe the mobilenet models or inception if you're not in a hurry.

Few questions:

1. Did you build this for your own use cases? Interesting side project?

2. How do you feel about the need for base64 being a requirement on the endpoints? Isn't GRPC the wrong medium for this? Also, what do you see as the main limitations right now? The models?

1. I built it to integrate with Home Assistant and security systems. I was trying to use Tensorflow on a Raspberry Pi and the dependencies were a nightmare. Tensorflow in general is a nightmare to compile and run IMO. I got to thinking, what if I could make all the deps inside of a docker container. What if I could run it remotely. It was born out of that.

2. As for base64, I'm not sure of a better way to support sending raw image data over JSON (in REST mode) In some ways I think GRPC is a better medium than JSON (it supports either) as GRPC supports sending the RAW bytes. What leads you to believe GRPC isn't the right transport? Plus you can do it in a stream format if you want to do a lot of video.

The only limitations I can think of are that Tensorflow supports a myriad of CPU optimizations so providing a single container image that has all the right options is basically impossible. I created one that has what I think are some of the better options (AVX, SSE4.X) and then an image that basically should run on any 64 bit intel compatible CPU. To get optimized options you need to build the docker container yourself which can take the better part of a day on slower CPUs.

With that said, I also provide ARM32 and ARM64 containers that actually run semi-okay on Raspberry Pis and and other ARM SBCs. I can run the inception model on a Pi4 on a 1080p image in about 5 seconds which is pretty good IMO.

> Nice writeup but the Raspberry Pi isn't running tensorflow. It is mentioned in the article that the author is sending images to an edge machine.

Yeah I was a bit surprised by this, and although the article is very clear about it I think it's generated a bit of confusion in the comments here. My understanding of edge computing is that it means the processing of data is done at the point the data is captured, so to me that would mean right there on the raspberry pi. But the author considers their whole LAN to be the "edge", so basically anything that doesn't involve sending the data over the internet:

> ... doing the heavy lifting on a machine physically close to the edge node – in this case, running the Tensorflow Object detection. By doing so, we avoid roundtrips over the internet, as well as having to pay for Cloud compute on e.g., AWS or GCP.

I think their strategy of capturing the data on a very low-power device and then processing on a server on your network is a very reasonable one, I just wouldn't have used that term.

This is where GStreamer normally steps in. A lot of hardware manufacturers provide a gstreamer plugin for their module. I’ve had experience with NVIDIA and atmel SoCs and that seemed to be the default path.

Good luck with the gstreamer pipeline learning curve however!

Can confirm not fun

Could you elaborate on some of the problems you had overall?

I've unsuccessfully dabbled in gstreamer in the past. I was doing a project this weekend, and the comments on this thread motivated to give it another shot .. after a couple of hours (2-4ish?), I was able to get video off the Pi to my desktop (on the same LAN) but the performance was pretty bad. I didn't optimize much yet but let me summarize the key issues I experienced with gstreamer these last few hours:

1) Very little documentation; poorly explained pipelines. I tried to read what docs I could find but things quickly devolved into trying out random gstreamer pipelines posted in comments. People don't explain why they use one particular element over another. So it felt like whack-a-mole.

2) Installing gstreamer on the Pi was a breeze. I wanted to pull video off the connected camera and sent to VLC on my desktop. Sounded like something that would work out-of-the-box? Nope. Kept seeing lots of stackoverflow comments of people stabbing in the dark, getting errors (or have the thing just sit there and not work) with very little feedback on what was wrong.

3) I have very little indication of what is hardware and what is software accelerated in my pipeline. I have no idea where latency is coming into my pipeline.

Overall .. my modern expectation for software frameworks is "batteries included" .. it is totally reasonable for sophisticated software tools to be complex .. but gstreamer is just not designed that way. While I got it to work, I see massive latency (likely because my pipeline is inefficient) and degraded quality (no idea why).

Curious what you found limiting about zeromq? Just not enough throughput for high FPS?

So far I’ve found it to be the sanest multicast solution since clients pull.

Issue isn't ZeroMQ. The simple/inefficient way to do it is to capture frames one at a time, and send them via ZeroMQ. Video is pretty bandwidth intensive .. the only reason things like YouTube work as smooth as they do is that they use codecs such H264/265 (which are proprietary unfortuantely) and stream compress frames over the network. Now doing the codec in software burns a lot of CPU as this is very math intensive .. most processors support hardware video codecs for this purpose. There are just no open source tools/libraries that make this good/simple enough that I have found.

Aren't IP cameras capable of H264 already encoded the video before output it to the network? H264 video stream has very good compression ratio and shouldn't consume too much bandwidth.

In the project I used zeromq, we were not using external IP cameras. My experience with IP cameras is still full of some seconds of latency .. I have no idea why.

Yeah I was hoping it was a raspberry pi maybe using one of these neural nets USB sticks, instead it was just using RPi as a dumb terminal for sending video. You could probably do the same with an old android phone set to stream video over lan.