Also most people don't buy a house for life, and if you sell the house through a realtor they get 6% or so of the proceeds.
This isn't something you can just logic an answer for without running it through a rent vs buy calculator, but investment expert and author William Bernstein suggests a rule of thumb that you should not spend more than fifteen years of market rent on a house purchase. In other words you can look up what the house you are considering buying would cost to rent, and as a rule of thumb if the sale price is more than 15 years of rent it's a bad deal. This is obviously an imprecise rule of thumb since it doesn't account for variations in real estate tax per region, but it's a starting point if you don't have a good rent vs buy calculator.
Can you give a more detailed example? What time horizon are you picturing? After you pay off the house you lose the leverage advantage of returns on selling a house you bought with a mortgage vs selling stocks, but you also stop having to pay the mortgage at all while you'd still be renting, likely at a far higher market rate, so looking at the post-mortgage-payoff curves, it's hard for me to see where the invested down payment + (mortgage + tax + upkeep - minus - rent) returns would be expected to be high enough to make up for that.
(The more rational counterargument, though, would likely be around diversification? Better to have land + stocks then just one...)