Well... that's complicated.

On it's own, XEP-0363 (HTTP File Upload) and OMEMO both work well everywhere, even in my horribly overpatched Pidgin.

But if you put the two together, Conversations, for example, will encrypt the message and encrypt the upload as well.

My Pidgin then will handle the message well, but display an url starting with aesgcm:// leading to the still encrypted file.

The question is then: is the correct behaviour what of Conversations is doing - encrypt the message and upload as well? If yes, is OMEMO a requirement to HTTP File Upload or the other way around?

I feel like encryption before payload and OMEMO are 2 different layers like, say, TLS compression and HTTP compression.

When you send a pic in a conversation, the content-type of the resource is "image/*". It doesn't matter whether the image is encrypted, compressed on-the-fly, or whatever, the receiving end must "undo" all of it to present the intended resource to the user

Encrypting the message AND the upload seems a no-brainer to me, if you want to call it E2EE. If the upload would not be encrypted then the (admin of the) server running XEP-0363 HTTP File Upload could see the contents.

I agree. The part I don't see is then what plugin should handle which part, and how.

Most XMPP clients are plugin based as well. Should the OMEMO plugin then look for aesgcm:// urls, download it, and decrypt it, or should the http upload plugin look for the availability of encryption and try to decrypt?

That is only a question if HTTP upload and OMEMO are different plugins. With https://dino.im/ it also "just works" and as a user I do not care which part pf the program does what.

Client authors should do what Conversations do. End users should probably stick with Conversations, for now.

There will always be hacky and badly implemented clients. It comes with the idea of an open specification. That doesn't mean we should use them as a point of comparison.