(And yes, I don't understand what most people want to hear either. Mostly I think they want others to compliment them, ask them to expound on themselves, and laugh at inane jokes.)
Your long-ish post implies that you prefer discourse (the antithesis of small talk). I suspect you'd like to pose an idea and then exchange ideas on it. While that was popular before the age of TV (much less internet), conversation on topics that resonate and last for 5+ minutes is unusual today, especially verbally, and it's likely that few strangers respond well to it. People like to tell / hear stories about other people, not discuss ideas.
Like you (I suspect) I suffer small talk badly, though in recent years I've learned to cut back on delivering 'large talk'... hopefully before peoples' eyes glaze over.
The build up of these echo chambers and extremely brief communication patterns contributes greatly to polarizing people on all kinds of different issues. It's really terrifying to me.
I think there is a way to verify what people are thinking and that would be by asking them questions. As you point out later it has to be genuine and not a clinical encounter, and I'd also add not a self-serving one. If you genuinely want to understand what they are saying you would then be very present in the conversation rather than just trying to hear what you think you want or don't want to hear (ie: "I've got to found out if he/she's going to abandon me or break my heart" is a self-serving interest and will eventually have the opposite effect), but trying to understand what they are saying because you are curious about them and hold them in high esteem will go a far way and asking questions that help you try to understand their point of view.
Too many people are trying to either narrate the relationship or are trying to serve their own self-interests and apply way too much meaning to each interaction and if instead they saw an encounter as an opportunity to learn rather than to try to have it mean something personally I think they'd find a whole lot more success.
I'm not extremely experienced in social interaction but I've had to learn to be to be able to communicate better with my wife and this is what I've learned, my self-centered motives have prevented me from learning so much more from others and using curiosity and questions has been super helpful in helping me to overcome social anxiety and relationship conflict fears.
tldr; let go of your ego (don't take things personally) and try to learn what other people are saying by asking them genuine questions (rather than trying to do all the talking, you learn more this way).
What do you (or anyone else reading this) think of evolutionary psychology in which context such aspects of human behavior can be somewhat consistently explained?
I should also add that for me, and probably for most people like me, the process of figurimg out what people like and dont like is also partly a process of self discovery. Im not the kind of person thats in touch with himself. Discovering how your words impact other people will also teach you about how your mind, conciously or otherwise, reacts emotionally to the words of others. Overall ive been genuinely excited to learn about myself amd others and use that wisdom to help enjoy the presence of other people. For me its been a very productive process of growth and discovery. I think framing the problem of interpersonal relations within that context instead of the cringey, manipulative context of internet social tips really helped.