Foot. Horses are for the rich, and castle builders wouldn't have been rich. Well the head stone mason might have been special enough to ride the master's horse once in a while. A horse eats a lot of food (even on days when you don't use it!) Most people would have walked. Saddles were not near as advances these days. Wagons would have had terrible suspension (and the roads not very good) so even the wagon drivers would have walked. A human can walk farther in a day than a horse can. The horse can run fast with a heavy load, but otherwise isn't worth it.

Most likely they would have used oxen not a horse to haul the wagons.