I agree that there are sensory perceptions humans are capable of perceiving and labeling as colors that cannot be attributed to external physical phenomena, but those are largely artifacts of the way our brain processes signals. For example if you stare at a purple dot for some time, then look away, you'll perceive a yellow dot where there is no external set of photons corresponding to the wavelengths that normally trigger the sensation of yellow striking your retina.
This is just more explanation about how "yellowness" is a characteristic of our brains, not of the external world.
Or did you mean something other than what I'm referring to here? I think that for the vast bulk of humans, the vast bulk of the colors they perceive regularly are due to photons striking rods and cones in their eyes at various intensities, causing color sensations to occur in the brain. Do you think something else is happening?
You seem to understand how the eye works, and some neuroscience, so I don't understand how you can have the questions that you raise about whether we can build cameras that sense "color" instead of "light"
Of course we can. We can capture the signal sent through the optical nerve and then reproduce it as a stimulus which will make the brain “see” yellow color.
Besides, humans are capable of distinguishing literally millions of colors, of which just a tiny fraction can be attributed to measuring particular wavelengths (or, more accurately, particular energies of the incident photons). In that way the eye is different from the ear (which performs a kind of Fourier analysis of the sound wave).