The concern isn't only coercion, it's also vote buying (which was very common at least in the UK before the secret ballot). If you can take the money and then vote for someone else, retail vote purchase falls apart.

The existence of postal voting has already cracked this one open a bit, though.