How many people who perceive classical music as elitist actually have a personal experience of someone on the "inside" acting elitist and shunning them as "outsiders"?
(There's actually no reason nowadays why this published "popular" music shouldn't become a part of the literate canon that's researched academically etc. But that only adds to a very long list of individual composers and entire musical traditions that end up being broadly neglected by standard academic POV's. It's also a general problem wrt. the study of human art and culture.)
Sacred classical music was accessible to poor people in big cities. Secular classical music was less accessible though.
I wonder if the cost and competitiveness of the training has something to do with the problem. The people who can win auditions are in fact elite musicians, and are almost exclusively from affluent families.
I enjoy the challenging music too, in the jazz genre. Given that I'm not trying to make a living at it, I'm grateful for the fact that I can play for a tiny but appreciative and loyal audience. I think there are a lot of niche genres that are sustained largely by amateurs and hobbyists.
I myself just feel lucky that I can enjoy most of this "challenging" music without needing to care about the magnificently strict structure, or how the composer wanted to channel God or something. All I ask is that people not question my sanity on the grounds of my liking of this kind of music.