Patents can be used to restrict all four of your freedoms under the FSF definition of "free software". They have written about this problem for quite a long time[1,2]. The problem with patents for things like formats or protocols is that they make free software implementations potentially dangerous to use (users can be sued for violating a patent). That's why h264[3], MPEG-whatever, FLV, etc are all considered to be bad even though they have free software implementations. MP3 recently shedded its patent licenses[4], and the response was that every distribution finally packaged mp3-lame in their main repos.

GPLv2 didn't have a patent grant because it wasn't a very well-known issue in 1991. Apache 2.0 was the first free software license to have a patent grant (which unfortunately made it GPLv2 incompatible) but GPLv3 included a similar (though stronger) patent grant because of Apache 2.0.

While these may sound like theoretical problems, people have been sued over using GIF, MP3, etc before. Every time you install Firefox, it will download a free software binary from a Cisco server that does h264 decoding[5]. However, they cannot legally recompile the binaries themselves as it means that the patent grant no longer applies and Mozilla (or your distribution) could be sued -- even though the binary's source is BSD-licensed.

[1]: https://www.gnu.org/philosophy/fighting-software-patents.htm... [2]: https://www.gnu.org/philosophy/danger-of-software-patents.ht... [3]: https://www.fsf.org/licensing/h264-patent-license [4]: https://www.iis.fraunhofer.de/en/ff/amm/prod/audiocodec/audi... [5]: http://www.openh264.org/index.html