Serfs were often descended from freemen who had to sell their freedom. The lords did hoard land, but they did not necessarily take it by force. Usually they inherited it, or they bought it.