It's certainly true that managers are lazy and they would rather measure illusory proxies for work rather than work itself, but America also became a major economic engine of the world by exerting control over people's bodies without paying them. Even though that practice ended several generations ago, the attitude that equates "success" with "power over bodies" is still very common in America. Many states are now trying very hard (again) to control women's bodies with the force of law, to cite just one example.
The actual economic data doesn't support that. Slaves were never more than 17% of the population, slavery was only ever legal in certain regions, and use of slaves was mostly relegated to a subset of agricultural labor. Moreover, slavery was outlawed in 1865 and the United States would not be considered an "economic engine of the world" until at least the 1880s, and it would not be considered one for its agricultural output but for its industrialized economy.
Furthermore, countries in the new world (e.g. Haiti, Brazil, that practiced slavery for longer and on a larger scale failed to develop strong economies anywhere to the degree that the United States did, so the line from "had slavery in the past" to "became economically prosperous" is tenuous. In fact, slave labor (and adjacent systems like serfdom) historically tends to hamper economic growth, prevent industrialization, and stifle innovation.
I once taught a course and the first thing I was told was to look for a green glow on people's faces: it meant people were playing solitaire on the class computers instead of listening.
A whole lot of managers knows only the work equivalent of that: they've learnt simple proxies for actual management that let them think they know when people are working well from walking rooms (or having people reporting to them do so).
People working from home means they actually have to figure out how to track output, and that takes effort. Effort many managers are not used to.
I had to learn a whole lot to start managing remote and distributed teams. It takes away a whole lot of lazy shortcuts to management.
Lucky for me I was forced to deal with that many years ago, but it was uncomfortable and a big change, so I'm not at all surprised that many resist it now.