It does not make the calibration (or preparation of profiles for various light conditions) easy though. There's a need for a bunch of HW tooling that is somewhat expensive for a hobbyist. And there's a lot of work and algorithm design that needs to be done outside of ISP for controling the autofocus, autoexpsoure and detection of type lighting in the scene and profile switching. None of this is particularly trivial.

If you compare with some other manufacturer that gives us absolutely no docs, then yes having register level docs and user side source is really nice. But if you want to do something unconventional (like switch the whole damn thing off and bypass its latency) not knowing how it works internally is still a black box. For example there is an irq that fires when the first component of the isp(MI-memory interface) sees a vsync signal for a frame. We know it exists. We know how to enable/disable that irq. We can write code that does stuff when it fires. But what we don't know is is this vsync signal fired truly at the input of the ISP? It's output? Or perhaps between some specific components? It turns out that vsync fires not when the isp sees the very first vsync what the location of the registers suggests, but after all the processing is done at the output buffer (40ms later usually), but one wouldn't be able to tell without a lot of profiling. This amongst other things is what I call a black box.

However, compare it with let's say a raspberry pi, and at register level the documentation is much better (for the isp).

There are bunch of interrupt statuses around frame capture: RIS_FRAME_IN, RIS_FRAME, RIS_V_START.

Not sure why you'd decide to use V_START for deciding when the frame is in memory. VSYNC start is sent at the start of a frame and it's documented as just a interrupt that is directly triggered from the MIPI CSI sync signal in the datasheet.

quote: "The incoming hsync and vsync control signals from the camera are connected to interrupt logic. It is possible to trigger on these signals for an event triggered configuration during processing."

The documentation is spread over many PDFs, incl. from different venors who use the same ISP like NXP, who tend to document things in a bit more detail.

So at worst, it's a gray box. ;)

I worked with rk3566, but I believe rk3399 has the earlier version of the exact same isp. The version in rk3566 is 1.1. There are also versions 1.0 and just 1. All are documented in TRMs in the same way.

It's not usable for RK3399 version of the ISP. BTW the same/very similar ISP (to RK3399) is used in NXP SoC, and it has documentation there, too. (NXP docs tend to be better ;))

If it was just the ISP there should have been a way in the past to take the ISP blob from similar chipset phones in the past and just use it with an updated base no? But that by itself doesn't seem to be enough to get the cam working decently.

There are simple things like debayer, but even just talking about exposure and gain control based on illumination is a pretty complex subject in itself. First you need to define parameters when to change exposure and when to leave it alone and just use camera gain. Then you need a set of parameters for when to use analog gain and when to use digital gain as communicated to the sensor. Then you have another two sets of parameters one for adjusting gain on the ISP (as opossed to all the previous stuff that feeds in to the camera chip) and some ISPs I'm told also have more advanced builtin correction based on curves. All this and we haven't even touched on how you measure the Illumination of the scene. At the very least you need one profile for back-lit and one for front-lit scenes. Then you need parameters for how the isp decides which profile to apply automatically and so on.

Mind that this is just brightness, we haven't even started talking about color correction, white balance, denoise, sharpen and 13 other ISP blocks that are there. BTW, you need another profile for every resolution your camera outputs too.

I don't know a lot about different ISPs. I only worked with one, but it gave me enough of an understanding to know why there is no way one profile will work for two camera chips. Or one profile will not work with two different versions of ISP hw.

That's why I hope libcamera succeeds. They try to standardise all the isp control, but it is a huge task especially when every single isp is different.

Over time the open source crowd implemented the state of the art image processing techniques once, then pointed them at reverse engineered backend drivers for each model.

The result was better than what the manufacturers (especially HP, but also canon, etc) could do.

I'm hoping history repeats itself with these camera sensors.

It's great to have more people working on imx219 though:)

Regarding my imx219 code that pushes it to 120fps I haven't submitted it upstream because the way fps is set changed in v5 Linux kernels. I used a v4 driver as a starting point because many other features only worked on v4(such as full gpu encode/decode etc). Also I preferred my userland side to not have to calculate timing parameters as it would require a per device implementation.

So my 120fps imx219 driver is for linux kernel 4.19 named rk3566_bsp_kernel-4.19, but I do have plans to upstream my code when I make them compatible with newest.