I suppose a lot do, but really more along party lines than because of a specific prime minister. You could be right in that the personality and politics of a particular leader might have a disproportionate effect on swing voters.

Personally I think the characteristics of the local MP you actually vote for is not taken into sufficient consideration by most voters. That's the person that actually represents you, and actually gets to help chose the actual prime minister. Who they are as a person and their values does actually matter very much, yet most people who actually vote can't even name their local MP.