The issue is it's why are you trying to be reproducible. The best use case is proving authenticity: that the source code became the binary code as written, but we're so far away from that that it's not realistic.

My dream system would be CI which gives me a gigantic object graph and can sign the source code from the ground up for every single thing including the compiler, so when a change happens you can drill down to what changed, and what the diffs were.