If there's no word for the colour you're perceiving then surely you make one or transliterate one? That's going to tend to create linguistic enclaves but mutual intelligibility should survive.

You can place where I grew up in the UK to within about 20 miles based on the word we used for lunch. Like the difference between bagged lunch (en-us) and packed lunch (en-gb) I suppose.