In practice, this isn't really possible, though. The binary is usually going to be slightly different. In theory you could RE the differences and potentially disprove a backdoor, but it's not easy.

Also, it's not necessarily that hard to slip a very subtle backdoor into the source.