It very much does, as color is not a physical phenomenon. It's just perception - essentially a bunch of weights in your neural network that makes some internal conceptual sense. You can trivially go outside of those "physical limits" e.g. by exploiting temporal perception quirks to get impossible colors (not represented by any spectral combination in typical conditions).

The same can be said about all human perception in general, not just colors. Artists exploit this for centuries, and are trying to formalize this. Newer color appearance models try to precisely quantify perception effects to make better color appearance possible.