There is always a reason why <insert groundbreaking and game changing technology> has not been widely adopted.

If it was all upside, the established players, in this case home builders, would have already incorporated it to gain a competitive advantage.

I am guessing there are downsides such as cost of materials, expertise, mold, or something that keeps this from being an economical solution.

Passive house is a very high standard, and I do not think it is very practical to do it at large scale.

Aiming for something that can be net zero is probably more practical: building stringently up to a certain point, and then throw some PV panels to offset whatever expected energy use is remaining.