Also I don't think it always makes sense to get a smaller house. We got a smaller house and I kind of regret it now, since housing prices and mortgage rates have shot up so much it's now so much harder to upgrade (and the house we got was intended to be a starter home), and also we'd get so much more if we sold a larger house today than our house now.
For example, let's assume prices went up about 50% on average in the past five years for the sake of easy math (not too far from how much my home actually did go up after a reappraisal last year). Buy a house at $300k and it goes up 50% means you can sell it for $450k and make $150k out of it. But buy a $500k home and it goes up 50%, means you can sell it at $750k and made $250k out of it. Granted you're making higher payments that whole time as well but not as much as it went up.
That math doesn't make as much sense now, with prices starting to dip a bit and high mortgage rates muting demand somewhat, however.
I don't really understand what people are imagining here though - it's not like "entertaining space" is some gilded-age ballroom for most people making this choice, and the price difference matters if and only if there is an otherwise comparable house you can buy that meets your other constraints.