> A program is free software if the program's users have the four essential freedoms: [1]
> The freedom to run the program as you wish, for any purpose (freedom 0). > The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this. > The freedom to redistribute copies so you can help others (freedom 2). > The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.
The FSF twice says that having "source code is a precondition" for the freedoms the GPL seeks to protect. And that makes sense. If what you care about is allowing users to "study how the program works" and "change it so it does your computing as you wish," mere freedom to reverse-engineer binaries is insufficient.
Indeed, your interpretation makes the first two freedoms meaningless. It doesn't infringe copyright to reverse-engineer programs for study, or modify them for your own use. You don't need the GPL to enable those freedoms. Those two freedoms only make sense if you read them to mean that users should be able to study and modify the program in the "preferred form" for doing so (i.e. buildable source code).
Only because reciprocity is prevented by default through copyright law. In the state of nature, I could not prevent you from modifying my published software, and you couldn't stop me from taking your published modifications and using them, or distributing around. GPL is crafted to create an enclave supported by copyright law, in which things behave similarly to the desired "natural state".
As for source vs. binary thing, I feel that if there were no restrictions on binaries, there would also be no reason to not release source code in the first place. This makes the source code aspect important for real-world GPL, but not necessarily a core issue for a hypothetical copyright-free world.