But yes, the receiver does have to measure small time differences between received signals; but that just requires a local clock with high resolution, not one with low drift or low absolute error.
For example, if your local clock is a second off, but still measures time intervals to pretty high precision (say, 0.01% - losing a second every three years) that's probably not going to mess up your location fix very much. Especially when the problem is overconstrained as it usually is intentionally. Compare the theoretical minimum requirement of three visible satellites to make a measurement with the 4-12 satellites that the system guarantees will be in sight at any time.
Receiver receives time “noon” from GPS satellite 1 at receiver-local time T=0.
Receiver receives time “noon” from GPS satellite 2 at receiver-local time T=2 nanoseconds.
Receiver receives a couple more “noon”s from other satellites, then uses its receiver-local time differences to trilateral its location.