Not GP: but I assume the answer is "because GPS is in the modem, the closed source untrusted part could still be actively disclosing your location".

.. But the modem itself is broadcasting it's identity to multiple radio receivers. Is there enough of a difference wrt GPS positioning for the distinction to matter?

Sure, I agree it’s subtle. But it is: a) much higher resolution b) potentially to a specific third party who may not have access to cell network data. Like, say, an angry ex-spouse.

Aha! That is a valid concern indeed. Lesser than the usual "baseband has full access to whole system", but still significant.