I'd wager that the phone sends back "areas" rather than GPS coordinates. So rather than the phone reporting "My owner was at (lat,long) at 3:21PM!" it would instead report "My owner came within range of Starbucks' Wifi Hotspot at 3:19PM, and left at 3:23PM!"

In reality Apple has been doing exactly this type of thing since iOS 4 was deployed[1], so unfortunately the privacy problems are apparently very easy to sidestep.

[1] http://www.dailytech.com/Apple+is+Tracking+its+iPhone+iPad+U...