>if the binary doesn't match what whomever gets when they compile, there better be a reason for it.
True, but deterministic compiles are stupendously difficult in most cases. Which is improving slowly, but still isn't usually an option.
Debian is up to 92% deterministic builds, so not that hard:
https://tests.reproducible-builds.org/debian/reproducible.ht...
I mean, at the end of the day, it's very easy to verify - if the binary doesn't match what whomever gets when they compile, there better be a reason for it.
Regardless, I don't think this is the biggest problem facing open source.