Consider any long-lived complex library. Count the bugs fixed since initial release. That’s your baseline — you can expect that order of magnitude of bugs in the initial release of your rewrite. Perhaps a new language will eliminate some classes of bugs, so maybe you can cut that number in half or even to a quarter. But you need to come to terms with that number. Now add on the number of bugs due to compatibility. Image formats generally are not tightly spec’ed, meaning the de facto standard libraries become the de facto standard — that is, the library being rewritten.
Rewrites are a long road that starts off one step forward and three steps back. It could sill be worth it, depending on the case. But an effort to “rewrite in rust” that doesn’t account for this will fail.
My point was that the test suite you start with for your rewrite will be (or at least should be) a lot better than the test suite you had when you started writing the original code.
The spec will likely be better too. Also, you can learn a lot from the previous iteration.
These effects, plus the benefits of the new language, have the potential to reduce the bug count dramatically. Maybe even by an order of magnitude.
Having said that, you're certainly right that there are some steps back and the net cost or benefit is likely highly context-dependent.
It would be great to have more data on projects that have done this, to help people estimate the costs and benefits in their context.
Writing image decoders for several formats is hard but it's not so hard that a trillion dollar company can't write a good one.