You could also be super smart and also passionate about something that immediately leads to making lots of money--Steve Jobs, for instance, is undeniably passionate about the kinds of products Apple makes, and how well they make them.
Thinking that money is just about money is a middle-class mistake. Money is capital, capital is power. Money is the lever with which you move the world. Money is how things get done.
The usual counterexample is academia, that a tenured academic has the freedom to pursue their discipline. That's not true and it hasn't been true for years. Most academics are under constant pressure to produce research and bring in funding.
You're right to say that you just need to make enough money to pursue your passions, but unless your passions are hopelessly mediocre, 'enough' has a lot of zeroes.
This sounds suspiciously like No True Scotsman
I have found this to be far from true. The truly brilliant people I know follow their passions, and usually make just enough money to do so, spending the rest of their energy on their actual passions.
Even if your ratio is right, that they get only a tenth of the opportunities as entrepreneurs... they tend to land the right ones, make the correct one, and be very satisfied with their lives.