News for hackers. Stuff that matters.
Re:Jetson AGX Xavier by TaylorAlexander on Sunday May 03, 2020 @04:16PM (#23061535)
Well, first some clarification - "edge" means "on robot" versus something in the cloud. And the reason you do this is latency and connectivity.

I am designing a four wheel drive robot using the NVIDIA AGX Xavier [1] that will follow trails on its own or follow the operator on trails. You don't want your robot to lose cellular coverage and become useless. Even if you had coverage, there would be significant data usage as Rover uses four 4k cameras, which is about 30 megapixels (actually they max out at 13mp each or 52mp total). Constantly streaming that to the cloud would be very expensive on a metered internet connection. Even on a direct line the machine would saturate many broadband connections. Of course you can selectively stream but this makes things more complicated.

Latency is an issue. Imagine a self driving car that required a cloud connection. It's approaching an intersection and someone on a bicycle falls over near its path. Better send that sensor data to the cloud fast to determine how to act!

On my Rover robot it streams the cameras directly in to the GPU memory where it can be processed using ML without ever being copied through the CPU. It's super low latency and allows for robots that respond rapidly to their environment. Imagine trying to make a ping-pong playing robot with a cloud connection.

I am also designing a farming robot. [2] We don't expect any internet connection on farms!

[1] https://reboot.love/t/new-cameras-on-rover/ [2] https://www.twistedfields.com/technology

Edit: Don’t forget security! Streaming high resolution sensors over the cloud is a security nightmare.

Re:Jetson AGX Xavier by epmaybe on Sunday May 03, 2020 @04:41PM (#23061701)
This is a bit off topic, but I'm constantly looking at ways to efficiently stream 4K cameras live to local displays as well as remote displays at the highest framerate and resolution possible. How feasible would it be on the xavier to stream 2 4k cameras and display them on at least 2 4k screens? Extra points if you could do that and simultaneously upload to a streaming service, such as twitch.
Re:Jetson AGX Xavier by 01100011 on Sunday May 03, 2020 @05:52PM (#23062142)
Best bet is to take the raw video and run it through gstreamer. You should be able to setup a pipeline which displays the raw video locally while sending a compressed stream to the network. I'd bet that Nvidia has gstreamer modules which make use of their compression HW, so it might be possible. To be honest though, that's a lot of data, so I don't know how well dual 4k would work. You can always scale it down in gstreamer before you send it to the compression module.

You'll probably want to use the CSI-2 interfaces to connect the cameras, but that depends. CSI-2 was developed for cell phones and is hard to run over long distances. It's optimized for low-power and designed for very short interconnects. We had a ton of problems using it at the last company I worked for. I really wish there was a competing standard for embedded cameras.

Re:Jetson AGX Xavier by deeth_starr_v on Monday May 04, 2020 @08:53PM (#23075168)
Not on a Xavier but on a TX2 I was able to get a single reasonable 4K stream with NVENC and gstreamer. Quality wasn't super awesome since it's realtime encoding. We ended up using two TX2s (one for each stream). I'd probably go with two Xaviers for better quality or step down and do 2 higher quality 1080p streams.
Re:Jetson AGX Xavier by TaylorAlexander on Sunday May 03, 2020 @05:46PM (#23062109)
You can certainly do this using machine vision cameras. Either USB3, Gig Ethernet, or CSI interface (16cm max run length I believe). I forget how best to attach two displays to the Xavier but that’s seems doable.

I got my cameras from e-consystems and they’ve got some USB3 cameras that could do it. At least I’m pretty sure. My USB3 cameras just showed up and I haven’t tried them yet.

Re:Jetson AGX Xavier by epmaybe on Sunday May 03, 2020 @06:48PM (#23062529)
I actually have two econ systems cameras that I've been testing on my desktop/laptop, and it does all right but even then struggles at 4k resolutions.
Re:Jetson AGX Xavier by TaylorAlexander on Sunday May 03, 2020 @07:41PM (#23062826)
Interesting. I know e-consystems provided their own kernel modules for the Jetson which help with some of the camera streaming, and they provide a special binary of I think guvcview to view live camera output. That is for Xavier though so I don't know if they have something for just a regular computer.

The other thing to look at is using hardware transcoding in an NVIDIA GPU, a raw 4k feed from a 4k camera is huge, but transcoded to h.264 or better h.265 the footage is much more playable from disk in my experience. It may help with live footage. Here are some notes I made when setting up GPU transcoding.

Using xstack filter:

https://trac.ffmpeg.org/wiki/Create%20a%20mosaic%20out%20of%...

Using h265 (required to support the extra resolution of 4x4k) http://ntown.at/knowledgebase/cuda-gpu-accelerated-h264-h265...

https://devblogs.nvidia.com/nvidia-ffmpeg-transcoding-guide/

What is your monitor resolution? If you can't display the full resolution, I've found that using the nvidia encoder hardware to resize each 4k stream to 720p makes transcoding much faster. I've added my video conversion scripts to github so you can see how I've done that. https://github.com/tlalexander/rover_video_scripts

You can also contact e-consystems, as they seem eager to provide application support. Finally feel free to email me to the email in my profile, or better yet create an account and ask the question on my website http://reboot.love so other people can see our conversation and benefit from what we learn.

EDIT: I JUST saw that you meant one local display and one remote display. Sorry busy day. In that case the other poster mentioning gstreamer is spot on, and I believe econsystems has some gstreamer plug ins at least for some cameras, or maybe nvidia does...

Re:Jetson AGX Xavier by m463 on Sunday May 03, 2020 @05:14PM (#23061896)
I've wondered about this too.

I think the magic camera interconnect is CSI/CSI2 and it's not really flexible enough. You either have really short copper interconnects, or unavailable fiber interconnects.

What would be cool is if csi to ethernet were a thing. either low latency put-it-on-the-wire or compressed. I don't know, maybe it is. But make it a standard like rca jacks.

Re:Jetson AGX Xavier by manofmanysmiles on Sunday May 03, 2020 @06:21PM (#23062364)
You can buy kits that send video over coax including power for somewhat reasonable prices:

https://leopardimaging.com/product/nvidia-jetson-cameras/nvi...

I haven't tried them, but am considered them for project.

Re:Jetson AGX Xavier by achuwilson on Sunday May 03, 2020 @06:11PM (#23062270)
It is possible to extend CSI through good HDMI cables https://www.robotshop.com/en/arducam-csi-hdmi-cable-extensio...
Re:Jetson AGX Xavier by fizixer on Sunday May 03, 2020 @04:24PM (#23061589)
Edge means on-premise (on robot) as you said.

But 'edge,' as used in context of AI, is also a wink-and-a-nod that the device is inference-only (no learning, no training). The term "inference only" doesn't sound very marketing-friendly.

Re:Jetson AGX Xavier by my123 on Sunday May 03, 2020 @04:58PM (#23061803)
AGX Xavier can do training on device just fine - and run every CUDA workload. It's just not the fastest device at that, you'd prefer a desktop GPU if you can for such a purpose.
Re:Jetson AGX Xavier by michaelt on Sunday May 03, 2020 @06:07PM (#23062241)
I assume what fizixer means is, if you're making an Amazon-Alexa-type-thing, training 1 model on 1 million user's data will work better than 1 million models trained on 1 user's data each.

AFAIK the "Roomba learns the layout of your house" type of edge learning is generally done with SLAM rather than neural networks. There might be other applications for edge learning, of course.

Re:Jetson AGX Xavier by jgalt212 on Monday May 04, 2020 @08:16AM (#23067037)
Yes, but the GPU is needed for training so "we" do this on prem or in the cloud. But the actual prediction computations are fast. Do we really need an edge GPU for these?
Re:Jetson AGX Xavier by TaylorAlexander on Monday May 04, 2020 @03:21PM (#23072179)
Yes. An edge GPU is required to run inference on high resolution sensor data for many robotics applications. Running inference is faster than training but it’s not all “fast”. Inference on CPU can take many seconds with large image operations. Not useful for certain robotics applications.