My worry is when this gets used for something forensic.

I imagine this will tend to reproduce things in the dataset, e.g. up-scaling blurry text may look like fonts that it has memorized more than the original. Or upscaling a feather will provide details like the feather of more common birds. Or upscaling blured out numbers will pick some numbers at random [1].

We need to make sure people don't rely on these details, e.g. in courts, HR reviews, when reddit sleuths try and investigate an incident, when someone looks for cheating partners etc.

[1] https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...

Yeah, the "uncrop" joke from red dwarf doesn't seem like a joke anymore (https://www.dailymotion.com/video/x2qlmuy).

Except if it's used in security camera, it's going to be a disaster. Those things have super low resolution, and software will be cheaper than upgrading. And models have huge bias.

Tons of fun.

Yeah, so much of forensic "science" is notoriously flawed, that this seems like a likely addition the usual junk-science pantheon of "gunshot analysis, footprint analysis, hair comparison and bite mark comparison" - https://innocenceproject.org/forensic-science-problems-and-s...

Personally, I think one reason we still do this has a lot to do with detective shows being so ridiculously popular that people think that it's some sort of scientific process, when it's not. As a thought experiment, we'd probably have flat-earther-ism be the dominant belief if was like 15/20 broadcast TV shows are dedicated to glorifying flat-earther-ism. This means the most dangerous thing about "Super Resolution" is the public has already been "primed" with cop shows having the "enhance" feature.

You’re worrying about the wrong thing. In formal settings, the problem will be taken care of.

The bigger problem is informal settings. Propaganda, for one.

The way DNA testing is misused, I wouldn't be so sure about formal settings.

Good point! You have me reconsidering what I said.

Although you have already replied to the DNA comment, one must take care with bureaucracies always. They will drag on with some preconception and the citizen always loses.

> In formal settings, the problem will be taken care of

Forensic evidence has been and still is systematically abused:

> * a 2002 FBI re-examination of microscopic hair comparisons the agency’s scientists had performed in criminal cases, in which DNA testing revealed that 11 percent of hair samples found to match microscopically actually came from different individuals;

> * a 2004 National Research Council report, commissioned by the FBI, on bullet-lead evidence, which found that there was insufficient research and data to support drawing a definitive connection between two bullets based on compositional similarity of the lead they contain;

> * a 2005 report of an international committee established by the FBI to review the use of latent fingerprint evidence in the case of a terrorist bombing in Spain, in which the committee found that “confirmation bias”—the inclination to confirm a suspicion based on other grounds—contributed to a misidentification and improper detention; and

> * studies reported in 2009 and 2010 on bitemark evidence, which found that current procedures for comparing bitemarks are unable to reliably exclude or include a suspect as a potential biter.

> Beyond these kinds of shortfalls with respect to “reliable methods” in forensic feature-comparison disciplines, reviews have found that expert witnesses have often overstated the probative value of their evidence, going far beyond what the relevant science can justify.

(https://web.archive.org/web/20170120002449/https://www.white... page 16)

Even more:

* Tire and shoe prints: https://www.apmreports.org/story/2016/09/27/questionable-sci...

* Lie detector tests: https://en.wikipedia.org/wiki/Polygraph#Effectiveness

* Burn patterns: https://www.pbs.org/wgbh/frontline/article/forensic-tools-wh...

I am not worried about it this at all. It’s not likely to randomly incriminate a real person.

I am from Brazil.

On the first day of trials of deep-learning based facial recognition here, a random person was arrested because the algorithm confused that person with another one.

Even more stupid, is that the person with "outstanding warrant" was actually ALREADY in prison.

So yes, AI managed to arrest the same person, twice, one time the real person, one time a random look-alike.

Seems like a different situation - the facial recognition algo was wrong. Not the same as an AI resolving a face that resembles someone and then prosecuting that person on the basis of the image.

You realize that the idea is the same, right? AI made an incorrect determination and people ran with it.

You do understand the difference between arrest and prosecution? People are arrested on the basis of a weak mistaken identity all the time, no AI needed

I remember that time a guy's life got turned upside down because his fingerprints "matched" those found on a bomb. Despite the fact that he had no motive, disposition, or access - they were determined to convict, because to do otherwise would be to admit that fingerprint analysis doesn't enjoy the scientific foundation that DNA evidence does.

So yeah, real people have been harmed by bad matching algorithms.

Yes, it is: https://news.ycombinator.com/item?id=26504144

Ok that’s pretty far removed from a prosecutor relying on that image though

Imagine low-resolution face photo. Police found 3 men whose faces are similar to that photo. Now they're super-resolutioning that photo and suddenly first person looks very similar to it, while second and third persons do not. First person is in danger because of algorithm which was trained on some specific photos.

One can imagine a lot of future scenarios. In fact that describes is an entire genre of story writing known as “science fiction”

It isn't "science fiction" though, we have a mountain of examples that demonstrate how probabilistic AI fails when the training dataset doesn't perfectly align with the eventual application. It isn't restricted to AI either, there have been numerous studies showing that cross-race identification to be garbage data.

<Insert other race> all look the same: https://onlinelibrary.wiley.com/doi/abs/10.1002/acp.898

This bus is an ostrich: https://arxiv.org/abs/1312.6199

Racist autofocus: https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio...

Google thinks black people look like gorillas: https://www.wired.com/story/when-it-comes-to-gorillas-google...

And here is an argument to look forward to: should the training dataset be build to represent the general population, or the specific subgroup that the software would be most often encountering? Because that FBI UCR...

Out of curiosity, do you think forensic science methods in general are likely to randomly incriminate people?

Gosling will probably go to jail for a lot of crimes...

Photography has always been more artistry than reality. From film selection, lens choice, framing and cropping, color correction, "dodging" and "burning" details in our out it's always been a compromise between what the camera sees (which itself may not be reality if you're using physical in-front-of-the-lens filters or other distortive techniques) and what the photographer want to convey. That's one of the reasons that Photographic Journalism often holds itself to ethical limitations with regards to how a photo can be manipulated after it's been taken. I'm almost certain that this kind of thing would at least walk that fine line if not fall right on over it.

Never mind the manipulations which can occur even before the shutter is ever pressed— I'm having troubling finding links now, but I know there was controversy a few years ago about a picture of bombing rubble with an object (a dress form or something, maybe?) standing up in the middle of it, apparently by chance which created a very artistic contrast, but there were later questions about whether the photographer had in fact arranged the object in that way rather than discovering and capturing a preexisting scene.

EDIT: Still can't find it, but here's a list which includes a number of other wartime photos which are proven or suspected to have been staged in various ways: https://militaryhistorynow.com/2015/09/25/famous-fakes-10-ce...

Exactly. That is also why claiming to only do "SOOC JPG" is ill-informed. People claiming it don't understand the process: to even produce a JPG, there has to be a process of interpretation already taking place.

When I volunteered for a small newspaper I did my usual, sometimes significant, editing (Lightroom-level, not PS) and didn't see anything wrong with it (neither did they; of course edits look better than plain JPGs). But I can appreciate how this becomes much more important with increasing range. Imagine if Pete Souza spoiled 8 years' worth of Obama presidency imagery just because he edited them in some obscure way.

On one hand as Ansel Adams said "You don't take a photograph, you make it.", and that still holds true today. You're influencing the picture when you choose your camera, your lens, your sensitivity (ISO), your color mode, your white balance, your raw software, your editor, etc etc all the way from the camera to the print.

On the other hand, there certainly is a difference between working with the information (pixels) you've captured, and inventing information by either drawing on the image or creating new data.

This methodology falls squarely into the gray area between those two.

Yeah, although technically the goal behind things like 'super resolution' is to add data that is not there anymore but probably was there before.

i.e. you had the original scene that was captured by a digital camera (a lossy operation) and then saved as an image file (often also a lossy operation), and then a tool like this makes an educated guess as to what information was lost in the 1st and 2nd steps.

You have to demosaic things anyway (as mentioned in the article) so it's not like you can escape algorithmic fudging of details.

Anyone who edits photos in Lightroom understands that modern photography captures both reality and artistry.

My goal when editing photos is to make them more clearly express how I felt or how I saw. This is often quite divorced from what shows up on the back screen of my camera.

I think photography has always been about artistry. You are taking a 2-dimensional slice of a 4-dimensional world. Interpretation is inherent.

Ansel Adams was surely no stranger to post-processing.

https://photofocus.com/photography/a-look-inside-ansel-adams...

There is a difference between normal post-production (cropping, defining colors, adjusting brightness, removing noise, ...) and adding details that just aren‘t there in the raw image.

How so?

One involves geometric meaning (e.g. edges, Bayer artifacts, clipped highlights), and the other involves semantic meaning (e.g. sky, faces, eyes). Dodging and burning is in the latter category and is equally deceptive.

Noise removal isn't semantic meaning?

It’s also very useful to be able to take a tighter crop of an existing image without worrying about pixelation

That was true of film photography and development too

A good experiment (which I cannot do): take a true photo of somebody, reduce it in size and then “super-resolve” it. How much must you reduce in order to produce a different “person” (or a non-person).

You can also repeat this indefinitely: 1. Scale down 2. "Super-resolve" 3. Goto 1 In Germany, we have this game "stille post" where you whisper someone a phrase and then multiple children try to whisper the exact phrase to the next. Most of the time a completely different phrase comes out in the end.

In the US we also play the same game, but call it "Telephone."

And we call it "Chinese Whispers" in the UK

It's always been more like painting than anything resembling a truthful representation of reality. In fact, the first ever photograph [1] much more closely resembles a painting than anything else, which is ironic.

All advancements have simply given us more control in how painterly we render our photographs, but have never _really_ brought us closer to the truth.

[1]: https://en.wikipedia.org/wiki/History_of_photography#/media/...

It is sort of. Their argument is that by taking into account the subpixel pattern of the sensor they can actually extract more detail than is readily visible in the picture.

Basically you can imagine that the blue subpixel is always to the top-left of the pixel. If you shifted the blue down and right one half a pixel you would have a more "accurate" production. In this way you can add a new pixel with the blue value closer to the right spot, then interpolate the blue of the original.

Of course you can also do logic such as detecting lines of different lightness and applying those on top.

So yes, especially with their machine learning they are adding new detail, but that is also likely some detail that was already there, but could not be conveyed with with the lower resolution. I wonder how different this would be from the simple approach of realigning the subpixels on a higher-resolution image and interpolating the "missing" subpixels. This approach may look better but wouldn't add any data.

Ansel Adams answering a similar question https://youtu.be/Ml__B0l9GIs?t=1514

Presumably you go from picture to painting the moment you open photoshop.

Photography for record keeping/science and photography for aesthetics/artistry diverge long before you get to techniques like super resolution. Which is still a fuzzy boundary because anyone who has taken a picture including a sunny sky can tell you raw photos generally don't capture how it looks to your eyeballs.

Not true, there’s always far more data in an image than a quick glance can decipher, astronomy field obviously pioneered numerous methodologies over the decades (many of which biologists do more shittyly, speaking as a biologist who did so). In the end I’d argue a lot of these “super resolution” methods are just glorified dexonvolution, not that it’s a knock on these methods but apparently it’s not cool to call deconvolution deconvolution anymore.

Most are, but in biology/physics STED[1] and STORM are physics based methods for overcoming the diffraction limit[2]. STED is pure physics, no math/deconvolution/AI tricks.

[1] https://en.wikipedia.org/wiki/STED_microscopy

[2] https://en.wikipedia.org/wiki/Super-resolution_microscopy

They use extra tricks at the image capture level to supercharge how much information you can load into the captured images (and then decipher them), but the methods are still related at least in STORM - you’re effectively deconvolving lots of sparse images and then merging them! Gaussian fitting of point sources is literally dexonvolution right? You’re just estimating the psf as a 2d Gaussian!

I am not qualified to get too in the weeds on the physics, but 'Resolution' is... complicated. Usually, when we talk about resolution we are talking about the ability to distinguish two points.

The 'resolution limit' (Abbe diffraction limit [1]) is related to a few things, but practically by the wavelength of the excitation light and the numerical aperture (NA) of the lens (d = wavelength/2NA). When we (physicists/biologists) say 'super resolution', we mean resolving things smaller than what was previously possible based on the Abbe diffraction limit. So rather than only being able to resolve two points separated by a minimum of 174nm with a 488nm laser and a 1.4NA objective, we can resolve particles separated by as little as 40-70nm with STED (but it varies in practice).

STED does not accomplish this by estimating PSFs and fitting Gaussians, it uses a doughnut shaped depleting laser to force surrounding fluorescence sources to a 'depleted' state, and an excitation laser to excite a much smaller point in the middle of the depletion (see the doughnut in the STED wikipedia page, Stephen Hell and Thomas Klar won the Nobel Prize in Chemistry for this in 1999 [2].

I know PALM/STORM uses statistics, blinking fluorescence point sources, and long imaging times to build up a super resolution image based on the point sources and computational reconstruction.

Not as familiar with that one or SIM, but I know the "Pure physics/optics" folks I work with regard STED as the most pure physics based one that doesn't rely on fitting, deconvolution, or tricks (not that any of that is bad or wrong!).

[1] https://en.wikipedia.org/wiki/Diffraction-limited_system#The... [2] https://en.wikipedia.org/wiki/STED_microscopy

No these are not deconvolution at all. They are using AI to add information to the image.

Superresolution in consumer cameras is definitely adding detail to the image using prior information about the universe (hair has a pattern, what looks like an edge has to be sharper than what the image says, etc). This is definitely the questionable artsy aspect of this superres boom. But the question I answered was more specific which supposed how you can make up info which doesn’t exist, but that’s definitely not true either. Modern superres tech (especially the non-deep learning kind) can extract more info from the image if you systematically account for the psf of the camera, distortions etc.

What makes you so confident that this "AI" doesn't end up just doing deconvolution?

Because the images being enhanced are in focus