It has always been a sign of wealth, i.e. no need to grow crops in front of your castle because you can afford it.