I guess it really depends on your definition of irrelevant. Personally I have no problem with candidates who have a storied past, since I seriously doubt they have made as many ill-informed decisions as I have. For some reason though I have a hard time accepting Romney's refusal to release tax returns before 2010 as legit. Maybe I just relate better with the type of candidate that smokes pot than the type that abuses offshore accounts :P
If more people came to see that perfectly capable people are often not squeaky clean in largely irrelevant matters we would all be better off.
Imagine a country where politicians don't live in fear that the public might learn they sometimes spark a bong, or once cheated on their spouse, or like people of the same gender as themselves from time to time.