I think what he's talking about is what Sony calls Pixel Shift Multi Shooting. Other camera manufacturers do it too. https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7r...

Wow that is in fact exactly it. I'm actually quite impressed they're able to shift the sensor array by exactly one pixel (or even close to) since that's on the order of microns.

In fact the Olympus implementation of this feature moves the sensor by half a photosite diagonally. They do it by re-purposing the existing in-body stabilization mechanism to move the sensor around.

User 'twic' posted a link to a very interesting article that describes this and also explains the difference between photosites and pixels:

https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-o...

Piezo actuators can make very fine movements. You can get positioning systems that can adjust position by nanometers, even sub-nm [1].

Also, cameras have had stabilization systems for a while now; I would assume they need similar pixel-scale precision. Some cameras shift the lens, some shift the sensor, but either way they need to shift the image-on-sensor by a very small amount, and also do it very rapidly.

[1] For example: https://www.pro-lite.co.uk/File/psj_piezoelectric_nanopositi...

Some projector manufacturers are shifting a 4K DMD around 4 times to increase the refresh rate of a 1080P image 4x.

All of them are shifting either by 4 or by 2 pixels you don’t have native 4K DMDs yet afaik.

There are some, but they're crazy expensive

Pentax did it with the K70 back in 2016.

They also did it earlier in 2015 with Pentax K-3 II. Also Olympus OM-D E-M5 Mark II had a similar feature in 2015.

Most modern mirrorless cameras have such hardware (usually denoted "in-body image stabilization"). The "repurposing" of it to acquire multiple captures at various slight offsets and combine them intelligently is maybe a bit more recent, but still pretty common at this point.

https://en.wikipedia.org/wiki/Image_stabilization#Sensor-shi...

https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7r...

https://www.nikonimgsupport.com/na/NSG_article?articleNo=000...

https://www.canon.co.uk/pro/stories/8-stops-image-stabilizat...

I think the technique was used a lot prior in astronomy, where I encountered it the first time.

There is called stacking, and the subpixel offset comes naturally from path distortion in the atmosphere itself.

I remember using registax when the first batch of consumer telezoom point and click camera came out and got a 30x long exposure of the moon, with, well, mediocre results.

Google uses it in Google camera. They're not shifting the sensor themselves, but they take advantage of the camera shake users introduce by taking handheld photos.

https://ai.googleblog.com/2018/10/see-better-and-further-wit...

Yes I'm very familiar with this but this is just MISR i.e. purely a software solution (Peyman Milanfar is one of the original researchers associated with MISR). Fortunately elsewhere in this thread hardware implementations have been demonstrated.

There are hardware implementations referenced in the blog post, and in the linked published paper.

> In the early 2000s, Farsiu et al. [2006] and Gotoh and Okutomi [2004]formulated superresolution from arbitrary motion as an optimization problem that would be infeasible for interactive rates. Ben-Ezraet al. [2005] created a jitter camera prototype to do super-resolution using controlled subpixel detector shifts. This and other works inspired some commercial cameras (e.g.,Sony A6000,Pentax FF K1,Olympus OM-D E-M1orPanasonic Lumix DC-G9) to adopt multi-frame techniques, using controlled pixel shifting of the physical sensor. However, these approaches require the use of a tripod or a static scene.

https://arxiv.org/pdf/1905.03277.pdf

damn the Ben-Ezra paper is exactly what i imagined being the scope of work for my planned dissertation. guess it's a good thing i didn't pursue it lol.

You can also do it in software by just recording IMU with high precision time stamps (a lot of camera sensors have a gpio they can interrupt on even every scan line) and post-processing. There’s cool techniques where they can remove various issues in rolling shutter through this technique to get global-shutter like quality and removing camera-movement induced motion blur. I haven’t heard of it applied to super res but I don’t see why not. I think Google uses similar techniques to implement their software HDR solution which takes 3 back to back snapshots at different exposure levels and merges them.

Newer Sony mirrorless cameras embed gyroscope data into videos for further stabilization of the image when IBIS is not enough

Theoretically, with 30FPS cameras like Sony A1 and said gyroscope data, you can create super resolution images.

IIRC Olympus' some handheld super resolution modes use both shake and sensor shift to increase resolution.

Synthetic aperture radar [1][2] uses this principle. In that case the imaging system is fixed on a moving aircraft or satellite.

I think in general satellite imaging is a good place to look for such implementations, since they have a naturally and predictably moving imaging system.

1: https://en.wikipedia.org/wiki/Synthetic-aperture_radar

2: https://www.youtube.com/watch?v=u2bUKEi9It4

SAR is similar but not the same since there you're super-resolving in time rather than space. Also in that instance it's just conventional MISR since you're not driving the imaging system (more information is being passively collected as the targets passes).

I think it’s more useful to think in terms of how well sampled the observations are relative to the size of the output space. SISR is very undersampled, and MISR is oversampled. SAR reconstruction techniques can fall in either bucket.

SAR is similar to the video technique mentioned, agree it's not quite the same but if underlying assumptions hold still more estimate than "guess".

A major difference here is that SAR uses phase information, whereas to my knowledge optical techniques are not doing that.

Pentax DLSR have this feature too. The same motors that are used to control the sensor and prevent camera shake are used to offset the sensor slightly while multiple images are taken

Certain Hasselblad cameras have had this for a long time, and Pentax, Olympus, Panasonic, and Sony have it on various models, using the sensor shift image stabilization to implement it.

I just bought a camera that does it, so it definitely exists.

https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-o...

That's a really informative article; thanks for posting it.

I also have an Olympus E-M1 MkII, but I haven't tried the high resolution mode yet. You just gave me a TODO item!

I haven't tried it yet either. It really needs a tripod, and i don't have one.

Probably not exactly what he is talking about, but this also sounds similar to dithering. Where with repeated measurements and random noise you can statistically estimate the value of a signal below the quantization level.

Indeed, it includes things like the Drizzle algorithm that has been used by Hubble space telescope astronomers for a while: https://www.stsci.edu/ftp/science/hdf/combination/drizzle.ht...

The iPhone 12 Pro physically shifts the sensor for image stabilization, so precise positioning is possible.