What a sensible minimum size is depends on the use of course, but at some point, you're giving up a dedicated library/office, room to entertain, a spare bedroom, room to put up guests, storage, a nice kitchen, etc. You don't need all that of course and I'm not suggesting most people really benefit from having a 5,000 square foot house. But at some point, the house starts to seem cramped and you're not really saving all that much cost. Even 1500 square feet versus 400-500 square feet is enormous in terms of usability.