As long as you retain the original (which you probably should anyway, especially if you're converting to a lossy format or downloaded it that way in the first place) and can redo the conversion on demand, I don't expect that to be much of an issue.

A Shazam-esque audio fingerprinting approach could work here, too (though obviously IPFS won't give you that for free like it would a file hash). Might be troublesome for remixes, though.