Wearing a bra will actually make your boobs sag faster, not the other way around. It's a very common misconception among women, though, who apparently can't be bothered to look these things up before shoving "advice" down your throat that "everyone knows" is correct.
http://en.wikipedia.org/wiki/Ptosis_%28breasts%29#Bras_don.2...
It turns out that just because something doesn't hurt, and looks good, doesn't mean it's good for you.
My naive, non-boob-having understanding is that using the wrong size will cause your boobs to sag faster, as well as causing other medical issues, even if it doesn't cause pain.