It's because we are building so many other things.
The price of a house depends on how many people there are to build it (like your grandfather).
These days people build all sorts of other things, so the price of a house goes up. But on the other hand you now have so many other cool things that weren't available back then.
And doing construction work, he paid for his family's house over the course of about five years. It wasn't a palace, but it was a nice house in a (then) nice neighborhood with a good-sized lawn. Today, people routinely spend twenty or thirty years to buy a decent house. Even dumpy houses in poor locations would likely take well over five years for many people to pay for.
Something seems very wrong.