I don't think college matters much as "job training" for the reasons you say.

The social hierarchy stuff is important though. If none of the credentialist ninnies running the country have ever lived in rural or working class places, well, there's eventually going to be a big disconnect. Wasn't always so; guys like George Kennan were from humble people and places.