Hah, thank you for summarizing this so well. I never really connected the dots on “visible” = “can be observed”.

indeed, every time i see such articles i remember my high school teachers’ metaphor: in science when we “observe” a phenomena doesn’t necessarily mean that we see it literally, like when we drop a pen to the floor and we hear the sound, we know a pen has fallen to the floor although we haven’t actually seen it with our eyes

There is actually an inverse correlation between the directness of an observation and the quality of the data. The human brain and sensory system are remarkable, but also limited and easily fooled. It is optimized for detecting food and predators, and reading the facial expressions of other humans, not making scientific observations. This is why, when accuracy matters, we build machines to do the work of observing for us.

Thank you for pointing out the distinction between "direct" observation through human senses, and machine-mediated observation for accurate scientific measurements.

It clarifies the motivation for extension/augmentation of human perception through technology, from microscopes to telescopes - and spectacles/glasses for that matter.

It also implies a continuation of the trend, where more and more of human perception will be mediated by technology, to benefit from advantages like accuracy, long-term memory/storage for data collection and statistics.

I guess augmented/virtual reality is part of it, where we can now perceive and experience a fully technology-mediated world, both real and imaginary. I picture sci-fi possibilities, like being able to "zoom in" to see things at magnifications beyond human biological limits - all controlled by thoughts.

This is hacker news; have you never encountered the visibility of identifiers in a scope? Or the visibility of side effects of a body of code? Visibility of a host on a network? ...

Virtually no distant astronomical image is a normal photo that corresponds to what the human eye would see.

They're multi-layer stacks of CCD observations filtered by frequency range, with false colors then assigned, not three RGB layers. IIRC, typically there's 7 - 11 ranges, depending on the instrument.

So one of the layers included in the depiction might be infrared, then false-colored into something the eye can handle.

Source: I worked in remote sensing.

Which file formats did you use?

One of my frustrations with ML is that almost no one seems to be training on HDR data. It's all 8-bit RGB. There is a big opportunity here to train GANs on raw sensor data.

But that raises the question of which file formats to focus on. Since you have lots of image processing experience, I'm very interested in what the data formats were like at your job. Was it proprietary formats? If so, were there any advantages over e.g. the RAW file format? Is RAW sufficient for standardizing "everything necessary to turn photon readings into a photograph that appeals to the human eye"? E.g. if I gather thousands of RAW photos, do I need any other information not present in the RAW file?

Mainly I am interested in any insights you gained on the job: you might know of some unique advantage that certain data formats have over others. I'm extremely interested in training ML models to synthesize what comes into a camera (photon readings), as opposed to what the world looks like to the human eye (RGB values).

(I apologize for not phrasing my questions more carefully. It's late. The essence is that right now, most ML training operate in terms of RGB, yet there seems to be many advantages to operating in "photon space". The trouble is, it's effortless to gather millions of RGB photos -- but it seems very difficult to gather a million RAW photos. I'm hoping someone knows of a massive trove of "some kind of raw data that can be turned into RGB values" – maybe that's RAW, maybe it's something else. Whatever it is, once I have it, I can feed it through an ML model.)

RAW from most cameras isn't "real RAW". Firstly it generally uses lossy compression secondly a lot of cameras apply some minor corrections to "RAW" data as well (e.g. noise reduction). Some would argue that most cameras can't give you "real" RAW since most sensors perform noise reduction one way or the other internally (e.g. most Sony sensors use correlated double sampling). Most sensors use analog PGAs to amplify the pixel signals before A/D conversion (that's the ISO setting)[2], which changes the characteristics of the sensor quite drastically (not just dynamic range and noise, a lot of older sensors had severe issues with tinting and banding at high gains). RAW formats are generally linear (those used for moving pictures are sometimes log-encoded, which makes more sense [1], but isn't "raw").

A library like libraw reads most RAW files and can give you an appropiately (for the sensor used) debayered RGB image. These images aren't in a defined color space; they're specific to every camera/sensor.

In the moving pictures space OpenEXR is a common format for "arbitrary" images, since it allows any number of channels each with different formats (e.g. you can have a file with 27 channels per pixel, some of which are float and some of which are integer).

[1] E.g. 12 bit log encoding gives even resolution across more of the dynamic range; a linear encoding gives half of each range to the brighter stop.

[2] Sensors that don't have PGAs simply have a fixed sensitivity. If the camera offers an ISO setting, it is emulated in the digital domain by pulling the image.

I’m assuming PGA is Programmable Gain Amplifier? It wasn’t immediately obvious from a web search.

Yes. The basic structure of a CMOS sensor looks a lot like a DRAM array, but instead of a capacitor you have a photo diode and the "bit lines" / column lines go into a programmable gain amplifier and then into a 12 or 14 bit ADC with some DSP [1] and a parallel-to-serial interface. The exposure is determined by the readout (row for row), that's what gives you rolling shutter.

Some sensors have global shutter instead, which makes it even more like DRAM: Conceptually each pixel has a capacitor and a photodiode, and a transistor connecting the two, all of these transistors are connected in parallel and form the "global shutter signal".

Some sensors have faster ADCs and use one ADC for a bunch of columns, which has been claimed as the source of column-banding in some sensors (unclear if correct).

[1] Sensors that are intended for both photo and video typically support things like pixel binning, where the sensor itself averages e.g. 2x2 blocks of pixels internally. A lower quality alternative is line skipping, where the sensor is told to only read out every nth line, thus reducing resolution considerably. The higher quality alternative is "full-sensor readout", i.e. the camera reads all pixels and downsamples the image to the video resolution. I believe some (announced?) sensors can do this in the sensor itself now.

Note: A lot of this is "somewhat informed speculation" on my part, because image sensor manufacturers tend to be very secretive of their sensor's details.

Annnnd comments like this are why I'm on HN daily. Thanks for sharing your knowledge!

TIFF is kind of standard for most scientific imaging, from astronomy to biomedical.

12- or 16-bit greyscale is common, with multiple channels for hyperspectral data. But since it can also store 32-bit data, floats, and complex numbers as well as rationals, it can be used as a container for pretty much anything you want to throw at it.

Regarding RAW. It's often used with colour CCDs with various types of CFA, like Bayer or CMY. It might have a higher bit depth, but it's still basically RGB but spatially separated and stored as greyscale. You could maybe feed it directly to your ML, but it's more common to filter it to create RGB data retaining the bit depth but at the expense of low pass filtering to remove aliasing artefacts. Or keep the separation to keep the high frequency components; really depends upon what information you want to extract.

No one in astronomy uses TIFFs. Most scientific images are stored in FITS format, either in integer form for some raw data or in floating-point form.

FITS is very domain-specific for astronomy. It's not at all common in the wider scientific community.

I used to work on on of the larger imaging tools, supporting over 150 scientific file formats. The vast majority, over 80%, were using TIFF as a container. The next most popular was HDF5. FITS was not something I saw commonly, and we were getting images from researchers all over the world.

If you want a terabyte or two of .CR2 (Canon’s “RAW” file format) files of astrophotography to train on, let me know. They were taken on a camera sensor that had a full spectrum IR conversion performed on it.

All I ask is that you share your findings with me and don’t redistribute the image data set without confirming with me.

Yes, absolutely! That sounds wonderful, and we certainly won't redistribute the data without asking.

We have a server with a 24TB NAS. If you send me your ~/.ssh/id_rsa.pub key, I can add you to it and you can upload straight to the NAS. Would that work?

I have an event to go to today, but if you hop into our ML discord server (https://github.com/shawwn/tpunicorn#ml-community) and DM me (I'm shawn), we can figure out the details. Or email me ([email protected]).

Thanks so much! This is exactly what I had in mind when I was writing my comment. I think it would be straightforward to extend StyleGAN to synthesize full spectrum data. And the results would look pretty darn cool.

(Not astronomy) When I worked in particle physics it was all TIFF, and often forks of libtiff with non-standard extensions, with nothing under version control. In hindsight it was a miracle the images were readable from one month to the next.

What problem you try to solve with GANs? Your RAW HDR photos are really taken by a sensor with RGB mask filtering out whatever is not visible spectrum. Scientific or indistrial imaging mostly use monochromatic information which is later colorized for human eyes.

Even if you get millions of RAW photos they will come from different sensors with different properties (noise, sensitivity, mosaic, orientation...) and you need to to normalize them all. sRGB is most common normalization target for which all prosumer cameras are optimized for.

The format is not really important here as it is usually array of pixels. You need more bits and resolution? Fine, but you also get more useless noise.

Color and dynamic range of most sensors extends greatly beyond sRGB, it's the camera's JPEG processing engine which maps this larger range to the smaller sRGB (or AdobeRGB) range.

You can attach custom ICC profile to JPEG, also to TIFF or PNG. But still you need to normalize data from different kind of sensors before you feed any ML engine with it. sRGB or AdobeRGB are most common color space profiles you can get with varied content.

Astronomy is done using FITS files[1]. Dedicated astronomical imaging instruments natively produce FITS files.

[1] https://en.wikipedia.org/wiki/FITS

Useful: https://arxiv.org/abs/1905.13189

Well, in remote sensing it was mostly 8-bit data per frequency range, then each satellite or instrument by name (LANDSAT, GOES, SPOT, etc.) has its own file format.

Sometimes a CCD bit was stuck, so we would process around that, etc.

The terms "HDR" and "photo" are not commonly used.

The Perkin-Elmer military system has fortran source, so all the file formats are documented there (if you can get access), as well as a minimal Scottish university system.

That, and the magnification. Sagittarius A* is 26,000 light years away (https://en.m.wikipedia.org/wiki/Sagittarius_A*).

That’s one and a half billion astronomical units. The star in question is twice the sun’s mass, and, I guess, not too dissimilar in size. If so, if you had a camera that could take a 1,000,000,000 by 1,000,000,000 pixel photo of our sun from the earth and pointed it at this star, that star would fill a single pixel (Likely a few more more due to imaging imperfections, but those would mean the light gets smeared out)

It typically takes multiple ‘photos’ to even get a single of those CD observations in a frequency range (https://en.wikipedia.org/wiki/S2_(star)#S0–102: “At one-sixteenth the brightness of S0–2, S0–102 was not initially recognized because it required many more years of observations to distinguish it from its local infrared background”

But if you were to use a teleporter to see these objects with your own eyes, they would match what is in the “photo”, right?

It’s not as though conventional cameras operate on the exact same principles as our eyes, either. And that’s without even getting into the amount of “computational photography” performed on phones.

> But if you were to use a teleporter to see these objects with your own eyes, they would match what is in the “photo”, right?

No. Very likely not.

The photos may be of wavelengths of light that the human eye can't see, so scientists shift (and stretch/shrink) those into the visible spectrum.

That means the photos could contain something you can't see with the naked eye.

If you were to teleport there, you would (probably) see something, but the colors likely wouldn't be like in the photo.

This video has a pretty good explanation of what's going on: https://www.youtube.com/watch?v=WSG0MnmUsEY

Many objects are too dim to see with the naked eye even if you were close enough to see them. Even amongst those which are clearly visible, they’d appear quite a bit dimmer than they do in a long exposure photograph.

The Orion Nebula is a good target for grokking the difference. A long exposure photo with a DLSR (which will be blurry if you don’t have tracking) you’ll see lots of color. Through a pair of binoculars, you won’t although you will see the nebula.