I think you're right, and the reason it's unaffordable is maintaining an automobile (even one that you don't race) is fundamentally very expensive both in time and money. Now consider the cost if you're pushing it to the actual breaking point all the time. Even lemons cars (which are supposed to be basically the least expensive way to go racing) are pretty expensive to keep going.

People are trying to address the pipeline problem you've highlighted by scouting talented drivers from simulators, which is great, but ultimately there's a huge cost to racing a physical car/kart/motorcycle compared to basically every other sport.