Where on earth did you find the "while everyone else is restricted to doing what I want them to do" part?

Libertarians who don't themselves do any drugs (eg, Penn Jillette) still want drugs to be legal for other people. Libertarians who are themselves pacifists still want gun ownership and self-defense to be legal. And so on. It seems to me that this view you think "falls apart" with so little thought doesn't have much to do with libertarianism.