This. An RSS feed of .mp3 or .ogg files is all I need for all of the formats mentioned by OP. It is annoying that some podcasts don't have RSS feeds -- this is what feels like a step back to me, but it's fundamentally a choice of the author/publisher. I'm much less likely to consume a podcast's backlog if it's not available as an RSS feed, and if the only obvious way to play content is through an embedded player or app that obscures the source media I probably won't play a single episode.

Edit: formatting fix.

If there isn't an RSS feed linking to audio files can you really call it a podcast?

Interesting question. I've seen collections of audio content claiming to be podcasts that don't have RSS feeds, so the term is definitely used that way. Whether that makes it a valid definition is open to interpretation, the word is pretty new still. Merriam-Webster seems to think that an automated download mechanism is required, though it doesn't specify RSS in particular:

>a program (as of music or talk) made available in digital format for automatic download over the Internet

I can see arguing that this doesn't even rule out walled-garden streams with an auto-play feature, since streaming and downloading are functionally the same (the only difference being what type of memory the data is stored in, and swap memory destroying that distinction entirely).