Typically these are called "modular homes" in the US. They've been around for quite a while and do make up a certain portion of the new home construction, though not the majority.

This approach is not quite as efficient as it initially appears:

First, it does nothing for the site construction (grading, excavation, utilities, paving, septic, well, etc.) Those tasks, which make up a decent portion of the project, are done in exactly the same way as in conventional construction.

Second, the modular boxes are either A) the width of shipping containers, which severely constrains the house layout, or B) somewhat wider than shipping containers, which makes them oversized loads that require escort vehicles, highway permits...increasing cost.

Third, you have to hire a crane to set the boxes, and then make all of the connections between the boxes, and install various mechanical and electrical components, complete the finishes, install staircases, etc. This costs quite a bit and still involves the hiring of various local subcontractors, just like in conventional construction.

When you add everything up, modular sometimes saves a little money and often reduces the project timeline (unless there are delays at the factory). But it's not a massive improvement, which is why modular hasn't taken over everything.

They depreciate in value like a car and have a stigma associated with the poor

That is true of "manufactured homes", a.k.a. "trailers", but not of "prefab" or "modular" homes, which are built using the same components as conventional homes.

They're the same components but not. If you would frame a building and then build the wall assemblies traditionally your ability to integrate parts of the structure and seal it is different than when you put together assemblies and try to seal the thick boundaries between them. You also don't have any quality control - whatever shows up on site is installed.

The loan to buy them also isn’t dramatically subsidized by the government and turning them into real property is a legal pain.

All house structures depreciate in value over time. The land they sit on is what appreciates, usually dramatically more than the depreciation of the structure. The prefabs associated with poverty are prefab structures on rented land, so the only thing you own is the depreciating structure, not the appreciating land.

It’s interesting - in Sweden, the vast majority of new homes are apparently prefab [1].

Aside from customer expectations for customizability in the US (and other countries) I believe regulations also contribute, with prefab homes actually requiring more robust construction than conventionally built homes.

[1] https://builtoffsite.com.au/emag/issue-05/sweden-became-home...

Does that make them cheaper/better?

Sorry, to clarify, I think US regulations require this, with the net result that prefab becomes more costly than conventional construction in the USA.