See the link I included on the opponent process. It explains how we have neurons that respond specifically to red/green and yellow/blue. That's why Y is psychologically as primary as the other 3.

Here's another way of looking at it: psychologically, we don't perceive yellow as a mixture of red and green, the way we perceive purple as a mixture of red and blue. Yellow isn't "perceived" as a mixture of anything in our minds -- it's perceived as primary. If you say "it's kind of a reddish-green", nobody is going to think, "oh you mean yellow!" While orange is perceived as a mixture of red and yellow, for example.

Remember, I'm not talking about what's happening physically with wavelengths -- I'm talking about psychological perception of colors.