who says the t-shirt you wear is/should be for the benefit of the random guy on the street?
Usually because random guys on the street see a lot more of the "logo area" of your t-shirt than you do (it's sort of outside of your regular visual field unless you're looking straight down or in a mirror). Though, admittedly, possibly-savvy coworkers see similar amounts of it. Depends which you spend more time around, I guess.
I 'got' the design because of the description, but if I had seen it randomly on the street, I'm not sure I would have immediately associated it with HN.
Not sure that any of this matters, but the marketing/branding side of me finds the whole color scheme thing an interesting discussion.