If GPLv4 were more restrictive, people would still be able to use the software under the terms of gplv2 or gplv3.

The scenario where FSF goes "rogue" would be if gplv4 became more permissive, e.g. if it stopped being a copyleft licence.