Re:K|Lens – The third dimension in image acquisition
by ricardobeat on Tuesday November 30, 2021 @04:27PM
(#29397689)
3-4 years ago we already had Lytro / the Light camera. Wasn't great apparently[1], but a much more practical idea than this massive lens.
Judging from the samples, the iPhone with ML-based depth sensing and Lidar also seems to do a much better job. As will most of the latest ML models available.
[1] https://www.theverge.com/circuitbreaker/2018/4/10/17218758/l...
Re:K|Lens – The third dimension in image acquisition
by echelon on Tuesday November 30, 2021 @12:20PM
(#29394639)
Light field tech is so exciting.
Light fields will be extremely conducive for building ML models around. It'll be possible to generate incredibly rich and realistic 3D scenes with the additional information light fields convey. Possibly some downsampled number of rays to make compute easier.
Multiple view scene constructions will make for incredible filmmaking that can be tweaked in post. No shot, reverse-shot. No cheating the camera. (That's assuming we don't switch from optics to mocap + rendering outright.)
First, why would you want lightfields? Computational photography -> variable boken (e.g. fix the bokeh for different depths, make it smoother, etc), depth info (extremely valuable for VFX), having some variability in your camera position - so being able to add camera shake more realistically. There's a bunch of other cool things you could do.
I've only taken a look at existing lightfield solutions briefly, but this looks cool. It's novel because it seems to only use the bare possible minimum number of ray samples to be called a lightfield, and makes up for it with heavy compute (even heavier than a typical lightfield).
This approach has interesting tradeoffs. A typical lightfield camera typically uses, say, 25 ray samples. This number of ray samples isn't enough to avoid compute, but cuts the resolution by a lot (25x in our case - instead of having 50 megapixels, you're at 2 megapixels). A typical implementation uses a micro-lens array in front of the camera sensor, and that means that there's wasted overhead due to tiling circles leaving a bunch of area empty.
Their proposed approach "only" leaves you with 9x less resolution (which is cool), which they try to reconstruct / upscale back to 2x less resolution (which is also cool and should be workable).