How are you addressing the idiosyncrasies of how clothing can fit differently? Personally, I like wearing a variety of different fits (slim/boxy/etc).

From your demo, it seems like the model is learning a single type of fit (the average of your training data?) and then mapping a texture + details onto this single fit.

In physical reality, the fit of a garment is obviously affected by both body type and garment measurements (as alluded to in other comments). Data quality (e.g., consistent scale and angle) also would be important. I imagine that because of these factors, the fit estimation problem is very challenging.

Best of luck with these interesting challenges!