It's a factor, but in a general election voters have to balance multiple considerations. From which local candidate is going to be a good advocate for my area, via which policy platform is best, all the way up to who should have the nuclear codes.

And then the candidate Prime Minister you (partially) based your decision on might resign, and the MPs of the governing party will pick a new person to exercise those powers, with no recourse to the electorate.

This is not theoretical. In the UK it's happened twice in the last six months.