Don't you think there is a representation of color in memory, that you can compare current visual input to? I can close my eyes and recall the image of a red apple, or are you saying my memory of red would switch to green as well?