When the mangaka is creating the layout, they're conceptualizing the flow not only of the panels, but also of the text inside panels, to be RTL.
Translating the text into a LTR language without mirroring the image, makes it so that your eyes have to zig zag around a bit more, going RTL panel wise, but LTR text box wise.
Compared to the problems that mirroring the art brings, I still think that's best compromise of the options, but doesn't mean it's not an actual impact on the experience, even if a subtle one.
I have wondered before, though, about how had might it be to learn to read mirrored, RLT english. Might be a bit of a challenge at first, but would enable you to read translated manga RTL with no compromises (other than the inherent lossiness of language translation in general).