People's second homes are generally in locations near tourist hotspots. That causes it's own problems but it's not usually prime economic centres.

Housing location is really important, not just the total number of vacant properties.