The confusion is on your end. Section 3 of GPLv2 requires you make the source available to any third party, not just your customer (and GPLv3's provisions are similar).
There is exactly one set of circumstances that permit you to say "no" to any and all non-customers, and it involves not limiting the initial distribution to just binaries. That is, iff you provide binaries together with the source to the customer who's giving out copies of the binaries themselves, then it is their responsibility and not yours, and only in that instance are you allowed to turn down third-party requests for source. This would also hinge on having a customer who would be fine giving out copies of the binaries they received from you in step 1 but for some reason wants not to give out copies of the source code that they received with it.
See also https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.en.htm...
> the Affero GPL only differs in that I would have to give you source code (edit: previously typed "software") if I supply a binary for you to use over a network
That doesn't sound right. You don't have to distribute a binary for the Affero GPL to require you to release your source. From the FSF:
> The GNU Affero General Public License is a modified version of the ordinary GNU GPL version 3. It has one added requirement: if you run a modified program on a server and let other users communicate with it there, your server must also allow them to download the source code corresponding to the modified version running there.
Edit (programmer needs sleep badly): Clearly it doesn't have to be a binary. It's any executable over a network. But you don't have to give the source cod to everyone, that's really my only point :-)
They are clearly consumer rights issues, to me. Whether you think they are important consumer rights issues is quite another matter ;-). Freedom 3, IMHO is definitely more important that Freedom 2. If you've managed to fix an issue, but are prevented from helping other people to fix the issue, I think it's problematic. As a consumer, I definitely value Freedom 2. Personally, I'm not sure I would consider it an ethical issue.
However, whether or not Freedom 2 is really an ethical issue, I think it's fairly straightforward to understand how people find Free software, as a whole, an ethical consumer rights issue. I'm not saying that everybody in the world is going to agree with the stance (and clearly many don't), but it's not much of a stretch to at least understand the position.
More edits for clarity. I should probably just go to sleep :-)