The whole thing boils down to one thing: whether something becomes a "derivative work" just due to being linked with code licensed under GPL. FSF says it does - and one could assume they know what they are talking about; after all it's their own license, right?
Except that the GPL license doesn't really say that; it's just FSF's interpretation. While e.g. the Mozilla license states very clearly what is and what is not covered by the license, thus avoiding the problem, the GPL is unclear.
There is, however, one very simple way to win any lawsuit that would claim that your binary blob violates the GPL: just make sure it's possible for you to demonstrate in court that your code can run with something else instead of the GPL thingie. If someone sued, say, NVidia, for putting their binary drivers into people's GNU kernels, they can point at the FreeBSD version of their driver: if it works without Linux, it clearly doesn't constitute a derived code; Linux is just one of the platforms it works with.
Some combination of the license (and existence of the LGPL), other things the FSF has written, practice, and history suggest that somewhere between statically linking your program together with GPL programs and shipping your program on a disk alongside unrelated GPL programs, you go from having to comply with the GPL and not having to comply with the GPL (for your own code).
However, much of the space between those two points is a matter of conjecture, legal theory, and opinion.