In terms of the colors of the world... I'm fine with nature... it's the man-made that gets to be garish at times.
I saw some pictures from a house where everything was white. The walls were white, the tables and chairs were white, the pillows were white, lamps were white, etc. But I bet the owner didn't see it as everything being white, like I did. I bet that when they'd look at the same pictures, they'd see all the little details here and there that were not white, or not quite white enough. I imagine them being in a constant state of stress over the non-white blemishes.
I don't know them, I just saw the pictures, so I don't truly know how they feel, I could be just imagining things. But my gut feeling is that someone with a home like that is not at ease with themselves.
I don't think I'd ever want something all that monochrome for myself. To me, subdued isn't about monochrome so much as limiting the noise. I don't even mind a sharp contrast, such as a colorful photo/painting. It's what I can only describe as visual noise that gets me. Especially with patterns that aren't really something you'd see in nature.
Once you step outside, it does matter though. If your own home is shades of grey, then any colour you encounter outside is going to seem garish.
Visually exhausting you say? If you are being stressed by the colours of the world, then that's a problem of your own making.