The elite special forces is a completely different ballgame. You need to be smarter and more fit than probably 90% of the population. Lots of these guys end up doing very well for themselves after leaving the military.
From personal experience I can say this is incorrect. The 'regular' special forces are the Green Berets (Top 5% of the Army). Elite special forces (CAG/Delta) 5% of the 5%
Not even 1% of the military can get into SOF, and that's less than 1% of general US population.
The people who get through BUD/S are generally Olympic class athletes who are selected for very specific mental and physical traits that are almost non-existent in the general population. There's absolutely nothing straightforward about getting into any SOF organization.