The way that I thought about color before being educated in the properties of light was to think of white as colorless. It's the thing that's easiest to add color to in a subtractive color model (e.g. using crayons paints as a child). It doesn't represent my science-based understanding, but I wouldn't be surprised to observe someone looking at something white and saying "Huh, it needs some color".