Sounds like you need a second container. I went for a full set of black socks and black underwear, and all anomalous socks/underwear that survived the QC process sit on the other side of a piece of cardboard I blu-tac'd to the draw. It works, and I find myself a fruitful object of comedy.
One caveat, though - if you do this, commit to it. Don't do what I did and keep one or two pairs of non-conforming socks just because I liked them. Because now they just recirculate through my sock drawer, gumming up my otherwise perfect system. :P