For early societies, you might be thinking of the Greeks, though apparently it's somewhat common between languages?[1] For Greek the idea to originate from Homer's writings, with colors being more described in terms of how light or dark they are versus specific hues.[2]

[1]https://en.wikipedia.org/wiki/Basic_Color_Terms

[2]https://en.wikipedia.org/wiki/Wine-dark_sea_(Homer)