That’s really not true, you needed a pretty large farm to have both the need and income for a horse.

The horse itself cost at least a third of a labourer’s yearly income (which was not really disposable), and that does not include a sufficiently large pasture or equivalent fodder.

Not only that, but those horses would have been work implements, like a tractor today, you would usually not get those horses out to go to church or see your family a village over. If only because most of the time the horse would be at work in the fields anyway.

Yup, individuals farms far away from anything is a New World thing. In Europe villages were pretty densely populated for most of history - usually about 0.1-0.5 km2 per family. You can't afford horses with that amount of farming land (and if you're a serf you can't technically own property and you aren't allowed to leave your village anyway).