And I’m sure the first peasants had some choice. Later ones ... not so much.
Peasants weren't exactly property like for example in the US, but in practice were bought and sold with the land (number of people living on a land was a large part of valuation) and could be mistreated or killed with no legal recourse.
Peasants, as opposed to slaves, were integrated into the civil society, although on very disadvantageous terms. Even on private land, they had their own judicial "councils" (look up 'sąd dominialny'), of course entirely on the mercy of the lord in practice. (There was also the state/crown land with marginally better conditions.)
All this didn't stray qualitatively from the broad Eastern European baseline. A Russian, Prussian or Hungarian noble wouldn't get punished for killing his peasant either, unless he'd seriously screw up his relations with the local regime. A French one pre-1789 would have to play his cards right. PLC got much flak in the period for peasant oppression because 1. it often didn't have raw political power to distract from it 2. there was the perceived hypocrisy in its political system, very loud about liberty, equality etc.
Except as I told, they did not made that choice. People who could raise the castle decided by themselves where they want castles. Castles were military and administrative buildings for actual powers.
You get to rule that land in exchange for set days of military service for king and in exchange of ensuring taxes are collected. Peasants opinions were not important.
> And I’m sure the first peasants had some choice. Later ones ... not so much.
Why are you sure?
Peasants did not had all that much say in that choice.