This is why there are separate commands like `icat` which will output ANSI escape sequences for rendering a PNG. Unfortunately a great many of these are terminal specific (iterm, terminology, kitty and VT200/VT300 (sixel) all have their own unique escape sequences. The closest to a standard we have is sixel and, frankly, I don't think it's that good on modern systems[1]). So you end up with different command for each "standard" you want rendered.
The approach I've taken in my own $SHELL is to have a builtin command like `cat` (I call it `open`) and that auto-detects which terminal emulator you're running and picks the right encoding to match what is supported by the terminal (if no standard can be auto-detected, then it falls back to ANSI art using coloured blocks). However the issue with that is now people need to run new custom shell instead Bash / Zsh / whatever. But the logic behind the auto-detection is really quite simple so that could be written it's own standalone tool.
[1] sixel re-encodes images in character blocks of 6 pixels. Where as iTerm, Kitty and Terminology send the image as is over the terminal via a base64 encoded string. All those solutions are inlined via ANSI escape sequences though.
This is not what I am talking about. I mean the binary headers inside png, jpg, tiff files, etc. These binary sequences can be easily interpreted as terminal control sequences, and they are very unlikely to appear otherwise. For pnm files I agree that the situation is a bit more delicate, as these sequences are likely to appear in the wild (e.g., in the pnm man page for example).
I'm very tempted to fork Aminal and have a play with your idea.
Notice that you do not even need to use sixels in your case. Since you control the rendering, you can simply put the pixels of the image on a rectangle (that allows paning and zooming, for example), and move the cursor below it.