I've lived in homes built that way. They're always a little janky, but usually much higher quality overall because they're built for a family and not for quick turnaround.
I know guys who are literally growing the timber for their dream home while they live in a singlewide. Lifestyle inflation in the USA is caused in large part by access to this easy debt, and it routinely ruins people in the bottom half of the income distribution.
There are other ways. Like buying one brick at a time. It's a real thing in places where access to lending is less common.