Your broader point is spot on however. My general hope is that people are wary when their results are strong^, ensuring that you don't get a good result via "cheating", so the majority of bugs are likely to harm performance. If a result is also not reproducible (i.e. "cheating" bug) it won't be used and built on - but if a result is bugged but reproducible (i.e. bug where performance was lower than it should be) then we can still move the field forward even in spite of these issues.
^ When I achieved state of the art for a task - especially given it was a huge jump in accuracy for a relatively small model compared to the previous state of the art - I spent many days sitting there double checking I hadn't accidentally cheated ;)
I'd like to imagine it's how effective peer review could be if given sufficient motivation ;)
As you note, though, most papers don't get anywhere in the same magnitude of focus, and others which do may still be entirely unreproducible anyway :(