Re:Libcamera v0.0.1
by mort96 on Tuesday October 18, 2022 @06:00PM
(#33254620)
To answer the first part: There's a lot of software out there which iterates through `/dev/videoX` devices, opens one of them (maybe by presenting the user with some UI to choose between them), and then just starts using it. You can query which pixel formats are supported and set one, you can query the range of supported frame rates and set a frame rate, you can query and set resolutions, etc. by interacting only with the /dev/videoX device you chose. However, this only works with some cameras; notably, it works with USB webcameras and most laptop webcameras.
But in the embedded/phone world, and with some recent laptops, this can't work. There is simply no single device which is "the" camera. The camera system is a complex pipeline of different image processing nodes. Here's how the graph looks on the hardware I usually work with: https://i.imgur.com/NSmu4Tj.png. The actual camera is the ov5645 node at the top, but that's not really useful in itself. You need to configure the media graph; in this case, I have set it up as ov5645 -> csiphy1 -> csid1 -> ispif1 -> vfe0_pix, and finally, /dev/video3 is a video4linux device (as opposed to a "subdev", which the others are) which an application can interact with using the video4linux IOCTLs. And importantly, you can't configure things like resolution, cropping, etc. on /dev/video3; you have to use the media control API to configure the ov5645 node's resolution, then you can additionally configure scaling and cropping and other forms of image processing on the vfe0_pix node. If you want to access your frames without doing any processing, you can instead link up the ispif1 node to a vfe0_rdiX node (RDI = Raw Dump Interface). My graph is relatively simple because only the vfe0_pix node does actual configurable image processing, but there's no reason the hardware couldn't be structured in a way where different nodes do different kinds of processing.
This is the stuff libcamera understands, and the stuff which normal "open a /dev/videoX device and throw ioctls at it" style application doesn't understand. The goal is for libcamera to figure out what graph it has to build to do what you want, to figure out what image processing operations are available at your various nodes, maybe apply various post-processing steps such as debayering if that's not handled by the image processing hardware, etc.
So the value of the compatibility layer is that an application which expects to just care about a /dev/videoX device can have its API calls intercepted, and libcamera will basically try to make it look to the application as if it's talking to a simple device where framerate/resolution/pixel format control/post processing/etc is all handled by the /dev/videoX device. In the background, libcamera will build the media graph, configure the graph nodes, do post processing, and whatever else is necessary to make the camera work. It basically makes a complex camera system look to the application as if it is a simple USB webcam.
I hope some of this makes the problem that's being solved here a bit more clear.
As to why people are hostile to it, I'd just chock that up to the general culture in Hacker News being of making fun of and demeaning other people's work, especially when they don't understand it.
But in the embedded/phone world, and with some recent laptops, this can't work. There is simply no single device which is "the" camera. The camera system is a complex pipeline of different image processing nodes. Here's how the graph looks on the hardware I usually work with: https://i.imgur.com/NSmu4Tj.png. The actual camera is the ov5645 node at the top, but that's not really useful in itself. You need to configure the media graph; in this case, I have set it up as ov5645 -> csiphy1 -> csid1 -> ispif1 -> vfe0_pix, and finally, /dev/video3 is a video4linux device (as opposed to a "subdev", which the others are) which an application can interact with using the video4linux IOCTLs. And importantly, you can't configure things like resolution, cropping, etc. on /dev/video3; you have to use the media control API to configure the ov5645 node's resolution, then you can additionally configure scaling and cropping and other forms of image processing on the vfe0_pix node. If you want to access your frames without doing any processing, you can instead link up the ispif1 node to a vfe0_rdiX node (RDI = Raw Dump Interface). My graph is relatively simple because only the vfe0_pix node does actual configurable image processing, but there's no reason the hardware couldn't be structured in a way where different nodes do different kinds of processing.
This is the stuff libcamera understands, and the stuff which normal "open a /dev/videoX device and throw ioctls at it" style application doesn't understand. The goal is for libcamera to figure out what graph it has to build to do what you want, to figure out what image processing operations are available at your various nodes, maybe apply various post-processing steps such as debayering if that's not handled by the image processing hardware, etc.
So the value of the compatibility layer is that an application which expects to just care about a /dev/videoX device can have its API calls intercepted, and libcamera will basically try to make it look to the application as if it's talking to a simple device where framerate/resolution/pixel format control/post processing/etc is all handled by the /dev/videoX device. In the background, libcamera will build the media graph, configure the graph nodes, do post processing, and whatever else is necessary to make the camera work. It basically makes a complex camera system look to the application as if it is a simple USB webcam.
I hope some of this makes the problem that's being solved here a bit more clear.
As to why people are hostile to it, I'd just chock that up to the general culture in Hacker News being of making fun of and demeaning other people's work, especially when they don't understand it.