That last clause is often overlooked or misunderstood, but it is critical.
It is, however, "do whatever you want" in that it implies no non-negative social duties, ie, it doesn't require you proactively do anything to be part of society or that you seek to help others in society -- merely that your harm to them come through passive actions or side-effects of legally permitted actions.
It might be better to explain libertarianism as "rights over duties", in that it essentially asserts you have no social obligation which is more fundamental than your rights.
Others disagree, obviously.
Any discussion can devolve to reductio-ad-absurdum if one is determined to do so.
But yes, that is the next most important division. Right libertarians believe that there is a natural right to property ownership. Therefore, infringement on that right is an aggression, and using violence to protect it is simple self-defense. Left libertarians believe that private property is a social construct, and that it has negative social effects at large scale (a few would even say that it has them at any scale), and so reject it, and the use of violence to protect it.
The other notion of freedom developed in the late Roman empire which came to mean a lord's right to do whatever he wants with his property, including his slaves. This notion of freedom, indistinguishable from power, lies at the heart of the Neo-Reaction movement.
https://www.opendemocracy.net/openeconomy/david-graeber/two-...
You're conflating laissez-faire or free market capitalism with a plutocracy (society ruled by the wealthiest). They're diametrically opposed systems.
Does that mean I'm not free to do pushups?
Does this clear it up for you?
Libertarians are nothing but outright hypocrites in arguing that government shouldn't be required to pay the economic damages of its coercing people to adhere to policy they favor (eg, maintaining a free-market, private-property based economy with courts enforcing contracts) while preaching about how people shouldn't be coerced.
Even in theory, libertarians are hypocrites because they fundamentally advocate for failing to pay for the externalities of their coercive system, and all supposed benefits of libertarianism can be traced to temporary boosts caused by (inefficient) capitalization of that failure to pay for externalities. Libertarianism enriches a few by creating 0.9 units wealth for every 1.0 units of collective suffering it implements over the current system, but hides this fundamentally externality driven mechanism behind the spread of the suffering (not a lot to everyone) and calling it freedom for many to be allowed to suffer at the hands of others without the instrument of state to implement collective relief.
Libertarianism is nothing but a sham political and economic movement that is wholly and fundamentally dishonest about its benefits and costs, by design, deeply self-inconsistent in its models and policies, and is intended to trick people in to accepting serfdom in a return to feudalism by calling it "freedom".
Libertarians, as a rule, just seem to have not thought very hard about their position or be selling you something (usually that you should be a serf because they fancy that they won't be a serf too in the new system).
However, the dishonesty of American Libertarians stems from something further than that. Namely, American Libertarians are neo-feudalists in practice, but won't own up to being neo-feudalists.
I find this to be dishonest: they, so far as I can tell, have done this purely because they want to advocate neo-feudalism and its benefits without having to justify the benefits in terms of the costs, so they use terminology where it's harder to find the costs in academic literature and deny that any such costs exist. Nothing different than creationism rebranding as "intelligent design" or anti-abortion as "pro-life".
Saying that libertarianism isn't about coercion and constraint is like saying that the GPL or copyleft in general isn't about copyright. Both are equally wrong: the GPL absolutely requires copyright law in order for it to work, and similarly libertarianism absolutely requires coercion and constraint.
It's just that there's this sleight of hand where for some reason, most people are so used to property right that they don't even notice it anymore. Talk about successful framing!
Capitalism is of course not an -ism at all. No one invented it. It's that freedom in the context of democratically agreed regulation whereby we can all trade goods and our labor with others and thereby generate value, usually monetary but by no means exclusively. We can even trade kind words and actions - they too contribute value.
Over the last century, billions of people have had the repertoire of 'things they can do' dramatically extended by the extension of an environment which allows people at all levels of income to take part in this productive exchange of items (abstract or physical) of value.
The very idea that people can have exclusive control of some goods and then have to trade for other ones is capitalism. Property is not the default state of the universe, it had to be invented and enforced before it meant anything.
And while it can be called a freedom in a certain sense, property is also a restriction. In order for me to own something, you have be denied the right to use it without my permission.
Which is not to say property rights are not a good thing on the whole, but they should be compared as one economic system, in terms of their outcomes, alongside the alternatives.