News for hackers. Stuff that matters.
Re:Super Resolution by ska on Thursday March 18, 2021 @01:01PM (#26504802)
One of those annoying things is that the name "super resolution" stuck, here.

Originally super-resolution was a hardware technique, and not "guessing". If you can [edit: this was poorly worded "control an imager positioning"] control imaging with finer resolution than the sensor has, you can take multiple images and reconstruct a higher resolution image in a principled way for say 2x resolution gain (cf super-resolution microscopy), also some telescope systems. Some modern photographic systems actually do this directly (piezo motors?) on the sensor.

Of course this only works if what you are imaging is reasonably static over the time needed to take all the images.

You can do an approximate version of this with video, with caveats because you don't control the motion. The key thing is, though, you actually have more data to work with.

This idea ran in parallel with image processing people attempting to estimate higher resolution from a single image for a while, and unfortunately the terminology stuck in image processing also. Something like resolution extrapolation is probably better but that ship sailed ages ago.

Re:Super Resolution by light_hue_1 on Thursday March 18, 2021 @07:02PM (#26508464)
This is the actual original meaning of the word "super resolution". The idea is that you use a prior about a specific image, domain, type of image, etc. to enhance the resolution.

Super-resolution was already the name for this field in this 1987 review! https://kmh-lanl.hansonhub.com/publications/imrecov.pdf But 1987 is pretty recent as far as this field is considered. A. W. Lohmann and D. P. Paris talked about super-resolution this way in 1964! https://www.osapublishing.org/ao/abstract.cfm?uri=ao-3-9-103...

So no. It's not annoying that super-resolution is the name of this field. The name predates the entire field of image processing, it predates the invention of the digital camera! And it is the original use of the title "super resolution", people that do this in hardware adopted the name later.

Re:Super Resolution by ska on Thursday March 18, 2021 @07:16PM (#26508579)
I think you misread me (so clearly I articulated poorly); the thing that is annoying is that super resolution in the sense you refer to is categorically a different thing that the common usage of super resolution in (particularly single image) digital processing. The usage existed, as you and I noted, before it came into digital image processioning.

Because of this, conversations around it are often confused, and it would be clearer if there were a different terminology for the latter.

Re:Super Resolution by meowface on Friday March 19, 2021 @03:10AM (#26510733)
What would be a better term for this type, then? "Inferred/approximated super resolution", or something?
Re:Super Resolution by vvanders on Friday March 19, 2021 @01:07AM (#26510316)
You can generalize this to signals by introducing noise to get higher resolutions from ADCs[1]. Human vision does this too[3], there was a really interesting theory that the reason 48 FPS in the Hobbit looked so bad was due to it falling right near the line Nyquist and the brain being unable to distinguish from a sequence of pictures(24FPS) or continuous frames(60FPS)[2].

[1] http://imajeenyus.com/electronics/20120908_improving_adc_res...

[2] https://accidentalscientist.com/2014/12/why-movies-look-weir...

[3] https://en.wikipedia.org/wiki/Ocular_tremor

Re:Super Resolution by formerly_proven on Friday March 19, 2021 @11:33AM (#26514510)
Doesn't strike me as true, 50 fps video is very common all over the world and doesn't have many issues besides noticeable flicker on CRTs.

Note that 24 fps film cinema runs at 72 Hz flicker due to triple-exposing each frame. That's why they could get away with 24 fps: Since they are dealing with stored images, multiple exposures were possible to avoid flickering. Using as low fps as possible was desirable because film costs money, film transports get more finicky as fps are increased and exposure gets tricky (doubling or even tripling the frame rate leaves you with a much shorter exposure time, which you need to compensate either with a larger aperture (which may not be available, or if it is, cost a lot of money), or more sensitive film (which may not have been available, or has much worse quality), or by having more light in the scene -- all of these are either undesirable or expensive).

So they chose 24 fps because the motion perception isn't quite in the shitter at that fps, though it's pretty bad (nowadays idolized as "cinematic look"; even more hip and cool if you do it on Youtube and deliver 24p which gets converted to 60 fps by means of 3:2 alternation, resulting in a super janky look known as "That Pro Youtuber Look"). This is the reason why people did not like The Hobbit.

Meanwhile video people went on to invent interlacing, because video was fundamentally about a non-stored image. Here the reason why low fps are desirable are again cost, since higher fps needs higher video bandwidth, and that makes everything more expensive and lets you have fewer channels (likely not a concern in the early days). Interlacing lets you have a higher frame rate (a CRT at 25 Hz or even 30 Hz will trigger nausea in everyone) without incurring higher costs, just like multiple exposures in a cinema projector.

Re:Super Resolution by Izkata on Thursday March 18, 2021 @07:55PM (#26508862)
> Originally super-resolution was a hardware technique, and not "guessing". If you can [edit: this was poorly worded "control an imager positioning"] control imaging with finer resolution than the sensor has, you can take multiple images and reconstruct a higher resolution image in a principled way for say 2x resolution gain (cf super-resolution microscopy), also some telescope systems.

Isn't this also how our eyes work?

Re:Super Resolution by Valgrim on Thursday March 18, 2021 @08:18PM (#26509008)
Not really. Our visual cortex does something closer to the new fancy photoshop algorithm (filling in imaginary detail from a mental model in memory). Super-resolution techniques such as the one described above is more like a mathemathical function based on a statistical values, so it extracts real information hidden in the difference between each image.
Re:Super Resolution by BearOso on Thursday March 18, 2021 @09:53PM (#26509500)
It does both. Your brain interpolates the image, but the eye also uses interferometry to generate extra detail by moving the eye back and forth really quickly (microtremors). The result is that we glean more true information than can be determined by just measuring rod or cone frequency.
Re:Super Resolution by anon_tor_12345 on Thursday March 18, 2021 @01:13PM (#26504983)
Citation? I've read over 50 papers in MISR and SISR (I wrote a lit review) and I have never seen mention of actual hardware that would shift the imaging system; indeed such a system would have been my dissertation topic if I hadn't switched areas.
Re:Super Resolution by galago on Thursday March 18, 2021 @01:19PM (#26505070)
I think what he's talking about is what Sony calls Pixel Shift Multi Shooting. Other camera manufacturers do it too. https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7r...
Re:Super Resolution by anon_tor_12345 on Thursday March 18, 2021 @01:42PM (#26505352)
Wow that is in fact exactly it. I'm actually quite impressed they're able to shift the sensor array by exactly one pixel (or even close to) since that's on the order of microns.
Re:Super Resolution by Stratoscope on Thursday March 18, 2021 @03:35PM (#26506593)
In fact the Olympus implementation of this feature moves the sensor by half a photosite diagonally. They do it by re-purposing the existing in-body stabilization mechanism to move the sensor around.

User 'twic' posted a link to a very interesting article that describes this and also explains the difference between photosites and pixels:

https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-o...

Re:Super Resolution by azornathogron on Thursday March 18, 2021 @05:36PM (#26507822)
Piezo actuators can make very fine movements. You can get positioning systems that can adjust position by nanometers, even sub-nm [1].

Also, cameras have had stabilization systems for a while now; I would assume they need similar pixel-scale precision. Some cameras shift the lens, some shift the sensor, but either way they need to shift the image-on-sensor by a very small amount, and also do it very rapidly.

[1] For example: https://www.pro-lite.co.uk/File/psj_piezoelectric_nanopositi...

Re:Super Resolution by RobLach on Thursday March 18, 2021 @04:10PM (#26507040)
Some projector manufacturers are shifting a 4K DMD around 4 times to increase the refresh rate of a 1080P image 4x.
Re:Super Resolution by dogma1138 on Thursday March 18, 2021 @07:22PM (#26508627)
All of them are shifting either by 4 or by 2 pixels you don’t have native 4K DMDs yet afaik.
Re:Super Resolution by bmicraft on Thursday March 18, 2021 @10:30PM (#26509680)
There are some, but they're crazy expensive
Re:Super Resolution by brudgers on Thursday March 18, 2021 @01:45PM (#26505383)
Pentax did it with the K70 back in 2016.
Re:Super Resolution by yfkar on Thursday March 18, 2021 @03:41PM (#26506683)
They also did it earlier in 2015 with Pentax K-3 II. Also Olympus OM-D E-M5 Mark II had a similar feature in 2015.
Re:Super Resolution by shock-value on Thursday March 18, 2021 @01:46PM (#26505393)
Most modern mirrorless cameras have such hardware (usually denoted "in-body image stabilization"). The "repurposing" of it to acquire multiple captures at various slight offsets and combine them intelligently is maybe a bit more recent, but still pretty common at this point.

https://en.wikipedia.org/wiki/Image_stabilization#Sensor-shi...

https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7r...

https://www.nikonimgsupport.com/na/NSG_article?articleNo=000...

https://www.canon.co.uk/pro/stories/8-stops-image-stabilizat...

Re:Super Resolution by avereveard on Thursday March 18, 2021 @03:46PM (#26506744)
I think the technique was used a lot prior in astronomy, where I encountered it the first time.

There is called stacking, and the subpixel offset comes naturally from path distortion in the atmosphere itself.

I remember using registax when the first batch of consumer telezoom point and click camera came out and got a 30x long exposure of the moon, with, well, mediocre results.

Re:Super Resolution by sorenjan on Thursday March 18, 2021 @02:50PM (#26506130)
Google uses it in Google camera. They're not shifting the sensor themselves, but they take advantage of the camera shake users introduce by taking handheld photos.

https://ai.googleblog.com/2018/10/see-better-and-further-wit...

Re:Super Resolution by anon_tor_12345 on Thursday March 18, 2021 @04:05PM (#26506985)
Yes I'm very familiar with this but this is just MISR i.e. purely a software solution (Peyman Milanfar is one of the original researchers associated with MISR). Fortunately elsewhere in this thread hardware implementations have been demonstrated.
Re:Super Resolution by sorenjan on Thursday March 18, 2021 @05:11PM (#26507602)
There are hardware implementations referenced in the blog post, and in the linked published paper.

> In the early 2000s, Farsiu et al. [2006] and Gotoh and Okutomi [2004]formulated superresolution from arbitrary motion as an optimization problem that would be infeasible for interactive rates. Ben-Ezraet al. [2005] created a jitter camera prototype to do super-resolution using controlled subpixel detector shifts. This and other works inspired some commercial cameras (e.g.,Sony A6000,Pentax FF K1,Olympus OM-D E-M1orPanasonic Lumix DC-G9) to adopt multi-frame techniques, using controlled pixel shifting of the physical sensor. However, these approaches require the use of a tripod or a static scene.

https://arxiv.org/pdf/1905.03277.pdf

Re:Super Resolution by anon_tor_12345 on Thursday March 18, 2021 @09:24PM (#26509364)
damn the Ben-Ezra paper is exactly what i imagined being the scope of work for my planned dissertation. guess it's a good thing i didn't pursue it lol.
Re:Super Resolution by vlovich123 on Thursday March 18, 2021 @02:45PM (#26506057)
You can also do it in software by just recording IMU with high precision time stamps (a lot of camera sensors have a gpio they can interrupt on even every scan line) and post-processing. There’s cool techniques where they can remove various issues in rolling shutter through this technique to get global-shutter like quality and removing camera-movement induced motion blur. I haven’t heard of it applied to super res but I don’t see why not. I think Google uses similar techniques to implement their software HDR solution which takes 3 back to back snapshots at different exposure levels and merges them.
Re:Super Resolution by bayindirh on Thursday March 18, 2021 @03:35PM (#26506596)
Newer Sony mirrorless cameras embed gyroscope data into videos for further stabilization of the image when IBIS is not enough

Theoretically, with 30FPS cameras like Sony A1 and said gyroscope data, you can create super resolution images.

IIRC Olympus' some handheld super resolution modes use both shake and sensor shift to increase resolution.

Re:Super Resolution by wongarsu on Thursday March 18, 2021 @01:39PM (#26505310)
Synthetic aperture radar [1][2] uses this principle. In that case the imaging system is fixed on a moving aircraft or satellite.

I think in general satellite imaging is a good place to look for such implementations, since they have a naturally and predictably moving imaging system.

1: https://en.wikipedia.org/wiki/Synthetic-aperture_radar

2: https://www.youtube.com/watch?v=u2bUKEi9It4

Re:Super Resolution by anon_tor_12345 on Thursday March 18, 2021 @01:46PM (#26505394)
SAR is similar but not the same since there you're super-resolving in time rather than space. Also in that instance it's just conventional MISR since you're not driving the imaging system (more information is being passively collected as the targets passes).
Re:Super Resolution by oivey on Thursday March 18, 2021 @02:25PM (#26505813)
I think it’s more useful to think in terms of how well sampled the observations are relative to the size of the output space. SISR is very undersampled, and MISR is oversampled. SAR reconstruction techniques can fall in either bucket.
Re:Super Resolution by ska on Thursday March 18, 2021 @01:58PM (#26505521)
SAR is similar to the video technique mentioned, agree it's not quite the same but if underlying assumptions hold still more estimate than "guess".
Re:Super Resolution by mNovak on Thursday March 18, 2021 @05:46PM (#26507895)
A major difference here is that SAR uses phase information, whereas to my knowledge optical techniques are not doing that.
Re:Super Resolution by gnopgnip on Thursday March 18, 2021 @05:50PM (#26507933)
Pentax DLSR have this feature too. The same motors that are used to control the sensor and prevent camera shake are used to offset the sensor slightly while multiple images are taken
Re:Super Resolution by CarVac on Thursday March 18, 2021 @01:18PM (#26505060)
Certain Hasselblad cameras have had this for a long time, and Pentax, Olympus, Panasonic, and Sony have it on various models, using the sensor shift image stabilization to implement it.
Re:Super Resolution by twic on Thursday March 18, 2021 @01:40PM (#26505331)
I just bought a camera that does it, so it definitely exists.

https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-o...

Re:Super Resolution by Stratoscope on Thursday March 18, 2021 @03:32PM (#26506555)
That's a really informative article; thanks for posting it.

I also have an Olympus E-M1 MkII, but I haven't tried the high resolution mode yet. You just gave me a TODO item!

Re:Super Resolution by twic on Friday March 19, 2021 @06:24AM (#26511768)
I haven't tried it yet either. It really needs a tripod, and i don't have one.
Re:Super Resolution by bobtail3 on Thursday March 18, 2021 @01:31PM (#26505241)
Probably not exactly what he is talking about, but this also sounds similar to dithering. Where with repeated measurements and random noise you can statistically estimate the value of a signal below the quantization level.
Re:Super Resolution by Thrymr on Thursday March 18, 2021 @01:40PM (#26505328)
Indeed, it includes things like the Drizzle algorithm that has been used by Hubble space telescope astronomers for a while: https://www.stsci.edu/ftp/science/hdf/combination/drizzle.ht...
Re:Super Resolution by dev_tty01 on Thursday March 18, 2021 @02:38PM (#26505960)
The iPhone 12 Pro physically shifts the sensor for image stabilization, so precise positioning is possible.
Re:Super Resolution by SiebenHeaven on Friday March 19, 2021 @12:28AM (#26510158)
A ASUS phone (zenfone 2) had this hardware superresolution feature you are talking about