Because you're thinking of homes in the modern term where one married couple owns one home and shares it with their children until they become adults and move out, instead of thinking of it as generational homes where 3-4 generations of the same family lived together.

So a master historically is not a 30-something year-old but his father in his 50s or 60s that actually owned the estate and wanted some privacy for him and his wife, hence a smaller, detached building away from the "main" building. The main building is where the master's sons and yet-unmarried daughters lived alongside the grandkids, hence needing more rooms, hence it being bigger than the building where the master lived.

I can't believe I have to explain something so basic.