When I was young I was exposed to adults and activities from many different backgrounds outside of school. More than helping me determine what I liked doing it helped me find out what I did not want to do. Things I thought I wanted to be based on what I had seen in books and movies turned out to be further from reality.
If you can gain these experiences it is a luxury. If everything matches up it is fortunate.
My suggestion is the larger problem is many workplaces are negative environments and leadership is lacking. The one real job I had was in maintenance. I did shit like mopping floors and bagging garbage. My boss was great. The people I worked with were happy. I was very happy. Once I started my own business it took me many, many years to be that happy again.
The number one piece of career advice for men should be not to get married or have kids. With enough money you can get a young trophy wife and have children at 55 no problem.
That's the thing I've never understood about following your passion. If you "can't not do" it, once people figure that out they will stop paying you, secure in the knowledge that you will keep doing it anyway.
Perhaps the secret to wealth is that your passion has to be money?
For instance: I can't not draw comics. I just finished spending four and a half years on a graphic novel. I like to draw them and I seem to be pretty decent at them if the reactions I've been getting are any guide.
But I have to pay rent. And power, and internet. And pay for my subscription to Adobe's Creative Suite, and upgrade my computer now and then.
If nobody's paying me to draw my comics, I'm going to need to find a way to pay those bills anyway. If I'm not lucky enough to have someone else pay them for me, then I need to get a job.
There are only so many hours in a day. A job will take up a lot of them. And I'll have to wedge tiny bits of making comics in the edges of what's left. I'll need to deal with a commute, and with spending all my energy at work, and just wanting to come home and sleep instead of draw.
So I draw my comics, put them out for free, and put a tip jar out with them. If enough people start throwing money in that tip jar on a regular basis, I can start cutting back on my day job. Eventually I can get to a point where I'm able to pay all my bills from comics, and don't need to worry about taking time from them to keep a roof over my head. I can spend my entire workday thinking about making my comics and drawing them, and get a lot more pages out a lot faster.
A lot of my fans are taking advantage of the fact that I'll draw comics anyway. But a subset of my fans have enough money to throw a few bucks my way and help ensure that I can make a lot more of the comics they like.
Does that make sense?
There's many things that make me who I am, things I can't not do, but if you're not going to pay me to do them, I will do them for myself and sell them to someone who will pay for them, or I will go do those things for someone who will pay for them.
I don't disagree with the secret to wealth being (or at least partly being) money though... those that don't care about the money are more likely not to be destined to have more than they need to provide what is important to them.
Or the passion is limited unique items (artworks) in which case they will buy it - if you become popular and supply is limited this can be self reinforcing like fine-art as investment.
Or they want the passionate person to work for them, so Google hire a passionate programmer so they can patent their work - knowing that the passionate programmer will improve in their off-time.
Or game companies who hire the young and passionate as they know they are invested in the end product and are less likely to bail if times are tough or they get a better offer (this can be exploitative and is closer to what you suggested).
Quite often things that become valuable are because people perceive qualities such as passion as 'honesty' in art and work this has value.
If people perceive someone is just in it for the money then they may less inclined to invest in their development as they could stop at any time and somehow the end-product seems less honest, a passionate person will do the best they possible can, some-one in it for the money will do just enough to get paid (hyperbole but it can be a perception).
True passion can be a very valuable quality - but it can also as you suggest be a crippling impediment to success, some are too passionate and unwilling to compromise so die poor ( e.g. Vincent van Gogh).
But it's still dependent on "your passion" being in demand in that sense, which is a fragile thing. There's a lot of demand for some musicians, and almost none for most of them.
Passion is what you care about. If getting a job done right keeps you up at night and you go the extra mile to do, then you have passion. If you can let it go then you don't.
I think better advice is: if you have a passion, then try to find a way to make it profitable if it's not and get into a position that you can act on your passion so it's not a detriment.
Having passion in an environment where it is not wanted is an awful experience and get a person fired really quick in an employment situation.
When a passion is profitable and works with others then it builds a fire that feeds itself and can be very fulfilling.
Everywhere you turn there are things that could be improved, fixed, wrongs righted, dictators toppled, famine, poverty, racism, sexism, classism, problems that need solving, things to be understood, things to be simplified; things that make everyone's lives safer, more bearable, more comfortable, happier.
If you take a hard look at the world around you and look at ways to apply what you enjoy doing to making people's lives around you better, your passion will come alive.
Why seek to control anything? Why not exploit it to make everyone's lives better?
For example, seeing your neighbor's abandoned, dilapidated house as a chance to start a farm or garden, is not something most people would think of when viewing the housing situation in Detroit. Grace Lee Boggs saw it that way but most do not.
The thing about control is about letting go of perfectionism. If you have an inclination to improve things, you can be overwhelmed by the number of things which could be improved, and it can be demoralizing to see the breadth of it all, and recognizing how little overall impact you, as one person, can have. So you limit your scope. You can't do it all.
Following your passion will make those problems worth the cost.
The problem is most people don't understand what "passion" is. Passion is what you can't not do (not to be confused with a warm fuzzy answer to "what is your passion?" which you're not really going to ever follow thru on even if handed every resource to).
ETA: as far as I can tell, few people actually have passions. If you don't have one, don't pretend to follow it. A career based on lying to yourself is indeed following horrible advice.