> the wording made it sounds creepy, like they can keep 
  > track of where I am whenever they have it, not just the 
  > destinations I ask for navigation to

This is actually the case, as far as I can tell. Check it out for yourself by going to Google Maps (doesn't matter if it's the web client or the app), open the hamburger menu from the button in the search box, then click "My Timeline". From there enter any date (in the app, first click the calendar icon) and it'll show you where Google knows your phone was at on that date.