> I wouldn't want to know a company when, how often and which doctors I consult for one.

Depends. A lot of doctor's offices are in "medical parks," so it's entirely possible they don't know which doctor you are seeing or why. They have easier access to that information via your calendar (if you use it) than your location.

Also, even if you did go to a doctor's office in the middle of nowhere with nothing else around and only one doctor working there, that doesn't mean you are there to see the doctor.

For that matter, your location data couldn't be proven to be yours on merit alone. Anyone can be using my car, and anyone can have my phone, at any given time.

You're underestimating the google, my friend. This response also misses the point, just because this specific example may not be relevant to you (what about private practice doctors?), there are hundreds of others that are.