Do we know that this is true for features of vision systems in the same way it is for the system in general?
For example, animals who end up in dark environments lose sight relatively quickly in a few generations, but for other cases, do animals tend to lose specific parts of their vision system - color, edge detection, etc.?
Yes, newts and salamanders in cave systems have lost their eyes, due to scarcity of nutrition and habitats devoid of light.
On the other side of the equation, vision is so stupidly useful that it has evolved independently from several different systems in different organisms.