Its a particular type of entertainment product that is completely divorced from its distribution method.
I could burn a podcast onto a CD and it would still be a podcast.
Would you say that a rich guy who has his assistants send emails is an email user? For example, it has been speculated that Trump has never sent an email.[1] But surely they didn't mean he's never had an assistant send an email. They mean he never used the application to do it himself.
[1]: Somewhere in this long discussion https://bloggingheads.tv/videos/58089
Language is flexible and user driven, but parts of how people use language is to defer to expert users for fine distinctions while making the broad categorizations themselves. Experts have determined what is and isn't a "monkey" but it's sort of an absurd category because it has to carve out homo sapiens but include old world and new world monkeys.
In this case, users might not know that podcasts have RSS behind them, but they certainly know that a podcast is prerecorded audio delivered over the internet and they somewhat defer to experts about the formatting, the way that they would agree that a "Betamax" is not a "VHS" but an "S-VHS" might be a "VHS" to a layman.
The word podcast in colloquial use has no inherent connection to RSS feeds.