Hi there!

So, all free/open multi-view stereo and structure-from-motion software I know of is incapable of handling rolling shutter artifacts.

The problem seems to be that electronic global shutter sensors lack (some) dynamic range compared to otherwise-equal rolling shutter cameras.

If you'd be interested in talking more about this, contact me/let me know (I'll get in touch if requested).

My photogrammetry experiments typically encompass low-effort bulk data collection, though the search for a light-weight camera to use with a prosumer-class drone and some revelations about reconstruction quality issues inherent to older, easily-optimized algorithms for both multi-view stereo and mesh reconstruction stalled progress somewhat.

In general, machine learning doesn't seem to be as much of a benefit as one might guess, when compared to applying the resources in non-ML ways to the data.

Mind teasing some numbers from your street capture?

Hmm and global shutter cameras are much more expensive. Though AFAIK a modern mirrorless camera with mechanical shutter can be used for this, yes? My GH5 can shoot at 12 frames per second with the mechanical shutter, so that may be of some use.

I did see in this paper that it seems like rolling shutter compensation is possible: https://research.fb.com/wp-content/uploads/2019/09/An-Integr...

I wonder if a library could be produced for this?

As far as my street capture I am currently using 830 frames from my 4 minute video (2 fps), images are 4992 x 3744 shot with a 12mm lens. The reconstruction is still texturing so I don't have the final result. What kind of numbers are you interested in?

Please contact me yes! My email is in my profile. See also my four wheel drive robot with four 4k cameras, and feel free to actually initiate discussion on that site where others can see it too (or emails is good). Thanks!

https://reboot.love/t/new-cameras-on-rover/

The mechanical shutter is not going to be that fast. About equal with top-of-the-line rolling shutters, except that the latter can sustain >100 fps.

I'm aware that global shutter cameras are more expensive, though I haven't even found a self-contained one that's notably below 500 g.

I'd like to know the polygon count, both vertices and faces, if possible.

I'll check the site and write more tomorrow, with the sun coming up shortly I should catch some sleep.

Also, yeah, rolling shutter can be compensated, but the impact goes through all the pipeline steps. All of them.

I don't want to re-write the whole pipeline. Worst case I'll settle for an Apertus Axiom Beta.

Also, a global shutter works nicely with a strobo, and a NIR-enhanced sensor works well with e.g. 808 nm light sources. The reason would be around power conservation when the robot has to bring it's own light source.

My last experiment had about 5k or so depth maps getting reconstructed (15 fps 4k Mavic Pro footage, iirc 100Mbit/s), from barely over 20k captured frames that I did full SfM on. The reconstruction density was adaptive, manually choosen to ensure sufficient scene coverage without spending too long on the depth map reconstruction.

The issue is that you have to assume somewhere between 1/60 and 1/500 exposure time for motion blur, if operating without an electronic global shutter. And I like detail, so I'm more looking for something like a 50 mm lens in front of an academy-sized sensor, yielding about 70~90 mm 35 mm-equivalent for the typical 4k by 2~3k sensor.

Reading your exchange I get the impression that a production camera would be suitable for photogrammetry?

Many older (and now inexpensive) Blackmagic models have a global shutter, and shoot RAW at 25 fps / 4k or higher.

Would you reccomend a wide angle lens for this purpose? Intuition tells me a telephoto would capture more detail, though with the caveat of requiring more movement over the surface area to be captured.

The first variable is what you want to capture. I am trying to capture large outdoor scenes in suitable detail to be used for robotics simulation. A telephoto would not work. I am preferring very wide lenses or even fish eye.

But some people are capturing individual rocks to be used for video game assets, or surveying historical sites or something else. There are times when a telephoto would be desired, especially if you cannot get close to the target scene.

4k is 8.5 megapixels so it’s relatively low detail compared to my 18 megapixel video from my Panasonic. I personally do not really know the math behind any of this so I am not sure where the trade offs are between high resolution rolling shutter or lower resolution global shutter.

Because I am doing scene capture for robot simulation, I do not need a perfectly accurate model. I would rather have a machine learning algorithm that places simple polygons where real world objects are in the video and gives them a reasonable texture. That is, I want to go from video directly to a low poly textured 3D model. My problem is complicated by the fact that I am developing an off road robot, and photogrammetry of forests is very hard.

This is why I mentioned machine learning. I care a great deal about the semantics of my scene and less about metric accuracy. What kills me is compute time. Finding feature points in every image and matching them all together is extremely compute intensive. I believe shortcuts can be discovered for problems like mine with creative neural networks. That said, I still need to learn more about existing algorithms.

Sure, if you get global shutter and resolution equivalent to about 5 MP bayer sensor or 2 MP monochrome sensor, and get output with an intra-only codec, that's fine.

Those black magic cameras are severely bandwidth bound, compare to e.g. the Pocket Cinema 4k (USB 3 and able to fully load a USB-C SSD), which unfortunately only offers a rolling shutter.

The lens is only slightly relevant, in that an extremely long focal length will cause issues in estimating said focal length, and typically also in depth-of-field if that's relevant for your scene. I suggest no ultra-wide-angle lens and no more than 500 mm (35mm-equivalent). 30-150 mm focal length (35mm-eu) would be the range you want to look at.

Could it be faster to skip in-camera encoding and build a 3D scene based on raw scene-referred data?

I think the far future of computer vision will be "event cameras" feeding directly into ML systems: https://en.wikipedia.org/wiki/Event_camera

It's not efficient to do all of this live. Skipping the H.265 parts would likely be beneficial, though.