A few millions are sufficient for you to be able to say "fuck you" to any employer or person you theoretically depend on, while still being able to live a comfortable live and - possibly - defend your right in court, even if it does not make economical sense to do so. That's what most people mean.

Having a few billion gives you some additional fuck you privileges - police officers, world leaders, other billionaires -, but you probably won't get on a level where you can say that to anyone and not fear repercussions. Also, if you can live happily on your own, you can already tell them that if you have a few million to your name.

In the US, for legal protection, I would say $10M is a good number based on legal costs I have seen. Maybe $5M works too, but I would say $5M protects you from healthcare costs and not being able to earn income due to health issues, but a little more is nice for paying lawyers if you are also not able to work at the same time.

There's always a bigger fish, but a few million is sufficient for almost every interaction most people will have in their lifetimes.

Unless you live in the US and get cancer.

This got downvoted, but I think is a legitimate concern. Having a terminal disease in the US can drain bank accounts quickly. It’s a real hazard that can destroy families / relationships.