I think you have a good point, but it is worth noting that being unable to communicate effectively significantly reduces your ability to change the world around you in a meaningful way. If you only have a few words for colors, then you're going to have a hard time exercising fine control over color-based problems.

However, one should also remember that language isn't just words. You can refer to colors by motion (pointing at a chart) or by proxy (using a code like #34F278). So we are limited by our language, but our language is not so limited as to make us fools for not having a word or two.