I often shoot with a histogram overlay which helps a lot, but there are still high-contrast scenes where I have to decide where I'm going to bias the exposure. I can usually tease out some more detail in Lightroom from the RAW file, but direct sunlight plus shadow, or a quick-draw moment where I fumbled the exposure to make the shot, often leave some clipped areas.
Not having to worry about clipping would be an amazing breakthrough, and I'd even accept some other limitations in a camera that could manage this.
If you're interested, try Tri-X for BW and Fuji Velvia 50 for color. Any late 70s Nikon will do well as a camera, the old 50mm f1.4 is particularly good and cheap. Tri-X is best developed yourself, but the Velvia needs sending to a lab that specializes in positive film development.
The actual headline of the article is actually more informative and less misleading than the subhead:
"Unbounded High Dynamic Range Photography Using a Modulo Camera"
In photographer's terms, they're referring to ending blown highlights (caused by filling the wells in a sensel) not saturated images (caused by exaggerating color data).
There's a school of photography that basically lives and dies by over-saturating images (e.g. http://kenrockwell.com ) and they'd likely have a cow over the headline, but understand and be totally onboard for the actual goal.
Depending on how quickly the well can reset and how many times resets can be counted this will either be better or worse than simply improving existing approaches. The state of the art in full frame sensors is just under 15-bits of dynamic range. So to equal it an 8-bit sensor would need to be able to count resets at least 128 times AND be able to do this in 1/8000 of a second, at the same sensel density and efficiency! Don't hold your breath.
http://www.dxomark.com/Cameras/Nikon/D810
(I'd also suggest that trends towards using multiple sensors to assemble high resolution images will easily blow this away since they can use high- and low- sensitivity sensors to synthesize more dynamic range and resolution. Also see recent Olympus patents to capture polarization data at the same time.)
BTW: assuming they can do this stuff, they presumably can read the sensor pretty darn fast — so they should be able to eliminate image "tearing" in digital video and the need for mechanical shutters altogether. That's probably a bigger issue than dynamic range.
I suspect that clearing a well in < 1/10,000,000 of a second (which would only lose 10% of the photon detection time during a 1/8000s exposure) is going to be tough. Assuming 25% of the sensel real estate needs to be sacrificed to the modulo circuitry, that's a loss of 32.5% of photon data which is about the same loss as for pellicle mirrors (as seen in Sony's pretty unsuccessful SLT cameras) for a feature that won't be much use in many situations.
Actually, I suspect that even that crowd, after the initial shock would be absolutely on board. This would mean that no longer would the sensor control how the high-end clipping happens, but now that could be tailored specifically via a post-processing filter. E.g. to exactly match a long-gone film emulsion's behavior, "enhance" a classic, or to simply to create new clipping functions.
This is a bit analogous to the flexibility that black and white shooters gain by working with modern color sensors. When shooting B&W film, the tonal result was controlled via a combination of the film's spectral response and any colored lens filters applied, the latter used to manipulate tonal contrast between different colored subject matter (e.g. sky, plants, skin tones, etc.). Now a skilled photographer can shoot color RAW w/ B&W preview (for a preview of the image luminosity) then adjust in post to nail the image effect without having to mess even thinking about what colored lens filter(s) to have used.
To emulate film stock we need a solid 14-bits of Dynamic range (while DxO says the D810 has it, the article below says 11-bits, but the best video cameras are getting there):
http://wolfcrow.com/blog/where-cameras-stand-in-dynamic-rang...
Ideally we'd get something like this AND capture polarization data as well (per the recent Olympus patent). Then you can apply polarization and exposure correction in post.
Assuming you have an accurate sensor that can be read non-destructively, then none of this modulus stuff is necessary anyway. Read once to set the proper gain on the ADC, then read again to get the fine bits. That way you still get the "unbounded HDR", but you don't need an ADC for every single receptor site, and you don't waste space on the counter or comparator. Instead, all of that lives on the image processor where it's not going to cause heat noise.
That's such an incredibly enormous hand-wave to me that I feel I've got to be missing something. Why make it so complex with all the modulus stuff if we have a magic sensor?
Oh the 1/10,000,000th comes from assuming they need to reset 128 * 8000 times per second to merely equal current sensor performance before losing sensitivity to misc losses.
If they want one more bit of DR they need to cycle in half as much time, and so on. Similarly, if we generously assume they're only losing 50% efficiency to cycle time and real estate then double again.
Seems like a better approach would be to add a "gain pixel" to traditional sensors.
So imagine you've got your Bayer grid (or a layout that functions similarly), one of the pixels in the grid gets read out first. That one sets the gain for the ADC for the surrounding pixels. The downside is your iteration couldn't be linear and the gain control would be a lot more complex, but you could do it with technology that actually exists, and you wouldn't lose half of your vertical resolution by doing it a whole line at a time like Magic Lantern's dual-ISO mode.
You'd have to do fancy processing around high-contrast edges, but what else is new.
The underlying idea is sound -- storing bits is a linear problem but storing photons is an exponential problem. Of course you're still going to have to dump exponential amounts of energy... (So forget the "unbounded" part.)
No system is ever unbounded, period, unless you live in a world of frictionless spherical cows. At some point you run into issues caused by accuracy, energy, capacity, etc.
I don't think it's a problem.
What's funny is that the iPhone (and a high end Android phone) has way faster processors than your typical $10.000 DSLR.
A good smartphone can capture 4K video or 20fps 8MP or whatever off the sensor — both around 160MB/s. A $10k DSLR is handling 16MP at 11fps — that's around 300MB/s (remember that the DSLR is handling 14-bits per sensel).
The interesting thing is that if you look across the camera lineups, there's almost no difference in CPU between a $300 consumer camera and a $10k professional (there is a big difference in RAM - the DSLR is able to store 50+ uncompressed images in RAM (let's say 3-4GB), while the phone or low end camera processes them into JPEG before storage.
What would truly solve the dynamic-range problem is a logarithmic readout CCD.
(Thanks for the Foveon pointer; it's an elegant approach to color.)
Not quite sure that's practical or how you'd implement it on a destructive-readout sensor, though. Right now the sensors with enough accuracy and sensitivity are all destructive.
Then again it doesn't matter, because that's a contrived example (the photography equivalent of a "microbenchmark").
The innovation with the tech in the article is in the dynamic range, current CMOS sensors have a fairly limited maximum dynamic range because of the well saturation described in the article, usually around 12-14ev DR. For high contrast situations that isn't enough, and photographers will definitely appreciate increased dynamic range, not to mention the other applications they listed, like astronomy.
On the display side most displays also have fairly limited DR and cannot reproduce the dynamic range in HDR images, so we end up having to tonemap images to get the perceived dynamic range back - however I think HDR monitors will break through in the next couple of years as many HDR TVs are hitting the market this year.
I work in post and have moved our workflow over to the Academy Color Encoding Specification (ACES) which supports up to 27 stops (more than the eye can see which is about 24 best case scenario). ACES allows us to dynamically change our output to rec709, rec2020, P3 etc and while 2020 has better colors, it's definitely still a highly compressed image (in terms of dynamic range).
Yep first "HDR" tv's will be rec709 but higher DR is definitely coming, both to home TVs and cinemas (via super bright laser projectors) http://blog.abelcine.com/2015/03/06/high-dynamic-range-expla...
I'm actually not convinced most people will want 5-100x brighter pictures than we currently have! Dynamic range of the human eye is also related to foveal/peripheral vision, so dynamic range in the real world probably doesn't map 1:1 to a small screen experience - if that makes sense.
http://www.rags-int-inc.com/PhotoTechStuff/CameraEye/
I am very interested in "natural", halo free tone mapping algorithms, squeezing that extended DR into current displays. I really like the results Adobe is getting with laplacian pyramids in Camera Raw for highlight/shadow recovery. Do you know of other tone mapping algos that look natural?
As for file formats, it varies entirely on the show. Higher end, fx heavy shows tend to use DPX and occasionally openEXR while lower end shows are typically some flavor of ProRes (422HQ and 4444 being nosy common). DNxHD 175 or 220 are also occasionally used. Some people are trying to move to the new ProRes high quality (4444XQ or DNxHR).
Delivery specs for whatever network are almost always ProRes 422HQ 720p or 1080p. Features we'll output to DPX or TIFF which will be converted to DCP (JPEG2000) in the P3 color space (XYZ instead RGB or YUV).
I'm excited to see higher bitrates for delivery, but it's going to be a long time on the broadcasters end. They're wary to upgrade so soon - especially when a lot of people already have trouble telling 720p from 1080p.
Even if there were, I pointed at the Foveon X3 as an example of a very good technology that was regardarded as a breakthrough but didn't really make it commercially. Another "revolutionary" idea that was marketed as something that "would change the way we take photographs" were the Lytro cameras, with their unique focus technology, that works by taking the picture in a different way than a classic camera, wich allows to change the focus of the picture working with the "raw" file. Even though that technology is also very interesting, it has remained in a niche market and only offered by one company -- more novelty than "mainstream" photography.
I don't think Lytro will ever take off in the way they intended, but light field capture will be very hot for 3D360 VR in a couple of years, when we start seeing first light field goggles (like Magic Leap)
16-bit ADCs increases tonal granularity in the readout, which can help in the darker tones. A sensor site may be capable of differentiating about 12 bits worth of meaningful information, hence there's a good argument for overshooting that in the ADC. Our eyes, even if not sensitive to more than 8 bits worth of tonal gradation, are perceiving in a logarithmic space (vs. the sensor recording in a linear space), hence extra information in the shadows can be quite good. Plus the extra bits give more room in which to tune a captured image. An article such as http://www.scantips.com/basics14.html gives a rundown on this.
The extra bits don't help if the incoming light exceeds the sensor's recording capacity, hence the interest of this "modulo" discussion for over-exposed images.
Is reading the sensor destructive to the data, somehow?
Assume we say an 8-bit image is taken every 0.1 second, if I merge 4 of them, wouldn’t I just get a 10-bit image with 0.4 second shutter time?
Because I essentially just take the RAW image, load them into a 10-bit colorspace, add them up, and then store the image.
Besides I have never seen "ordinary people" fumble with aperture size and exposure length so that's a solved problem. Sometimes it gives "wrong" results, like exposing for the highlights rather than the shadows, that's what this technology solves.
> there's simply no way to expand the camera sensor dynamic range without taking multiple exposures
That's the point of this paper†. They expand dynamic range by having the detectors wrap around (discarding high bits) and then recover the high bits computationally.† here http://web.media.mit.edu/~hangzhao/modulo.html (linked by putterson in another comment)
The only thing I would consider that solves "fumble with aperture size and exposure length" would be huge ISO range (both ways) without any noise.
> No more will photographers or even ordinary people have to fumble with aperture size and exposure length.
> I just don't see how it solves this issue.
What would normally have been indistinguishable blacks are pushed into the midtones and you get a flat, grainy image. It just looks like garbage because noise gets out of hand and you lose all your color depth, but there's more dynamic range there in the sense of there being more stops of light crammed onto the final image.
There is a physical limit of the camera or the film where you cannot actually squeeze any more accuracy out of the ADC, but standard conditions are nowhere near that limit.
This is a "computer scientists" understanding of photography, and this phrase alone can even be seen as "dangerous" by photographers. There's more to aperture size and exposure length than "how much light reaches the sensor", like focus, depth of field, motion blur and bokeh, to name a few, that is not aknowledged by that understanding. The technology is good, but it won't change the way photographers deal with photography, at best, it will give them a better tool to work with.
I also wonder how much impact this could have. Foveon Inc. had developed a great sensor that was going to be revolutionary, about 20 years ago. Today only Sigma uses it, and the UX is so bad in those cameras that the (very good) technology they had their hands on never got a chance to shine.
Also, is this implementation very different from having a 16-bit ADC in normal sensors? The end results would be the same: higher dynamic range. And the increase in bits seems a lot closer (because it would mean little changes to the current manufacture processes) than a whole new type of sensor being use in cameras.