I imagine anyone doing this seriously would have a spearate system for building binaries. Obviously if that system was compromised, you would not be in a great situation, but it should be much easire to lock that down.