Not just the location permission; apps have been found to scan pictures taken to build a location history out of the location metadata that is stored in pictures and such.

Practically speaking, unless you disable location tagging in pictures, any app with media access can track your coarse location history, depending on how many pictures you tend to take throughout the week.