Anecdotal evidence to be sure, but I went to an Ivy League School (Penn) on just straight-As and 700s (!) on the SATs. I was not a "legacy" student of any kind. I knew many such people there, and only about 10% of my colleagues admitted to being a legacy -- maybe it was just my social circle? Or that I didn't major at Wharton, which is stereotypically legacy?

Your point about graduate school is spot on, though. The game completely changes (though I think it's still mostly a game). I especially love the "don't buy this muffler..." quote!