Everything you say is true, but you can still do it if you're that way inclined. As long as you don't mind poverty and social exile :-)

Career-wise this has been a terrible life choice for me and I advise people to go to college for precisely the same reason you do. On the other hand career and material gain are only one aspect of life, I can't say with conviction that I really wish I'd done anything differently.