Wealthier residents generally avoid urban transit simply because they do not live in urban areas. Post white-flight, cities are generally quite a bit poorer than their surrounding suburbs. Cities that were built before that phenomenon (e.g. NYC), have good transit that even wealthier residents use.
Wealthier residents categorically avoid urban transit in many major municipalities because they are generally unsightly (dirty, littered with ads) at worst and generally unreliable (complete and total lack of consistency in travel times and distances from homes) at best.