Wouldn't it just make sense to tell them not to use GPS?

That's like saying just pretend the internet is down. The goal is to find unanticipated uses.

Also, "jamming" can include sending distorted signals. These distortions can be a spoofed or malformed packet of data. Knowing what happens when a navigation tool suddenly thinks the vehicle is on the other side of the world is important. As is, ensuring that a garbage signal won't crash it.

Or less extreme things. Do the soldiers notice when their GPS shows them as half a mile away from where they need to be. In a slowly changing offset.

That wouldn't test the software, assuming cheating wasn't an issue

Is “them” civilian pilots or the soldiers? Because the military may have all sorts of equipment that has black box GPS functionality they can’t easily control (think of GPS guided munitions for example).

You know how we established radio-comms in the military? By debugging over cell phone links obviously :-)