The Bakers, Smiths, and Shoemakers were in their own sense entrepreneurs. They ran their own businesses, kept their trade within their families, and profited strictly through the fruits of their own labors rather than a capitalists willingness to give them crumbs of their efforts. Owning your own small business, or having a family owned business, is incredibly divergent from the status quo of employment to international megacorp where everyone you ever meet involved with the company is still just a low level cog in a larger machine.
The Bakers et al saw their prosperity often strictly through their skill and talent. We have normalized that significantly in modern society for better and worse, but the inventive Smith who could improve on his metalcasting skill through ingenuity would reap tremendous benefits in the kingdom. Trying to innovate in shelf stocking at Walmart will get you fired. The ability to dictate ones own destiny like that, even if constrained to the skills passed down from parents, is still providing absolutely useful work being done. The Bakers and Shoemakers existed because of real demand from not only the nobility but their neighbors for their work.
There were few or no unemployed in the era of Smith families formation - if you did not have practical utility to your community you were often discarded. You prayed on the good nature of your own children to keep you around in old age when you were decrepit and could no longer work. Before the Industrial Revolution, the marginal cost of maintaining an individual persons working conditions was enormously high, such that 80% of people at least were working as farmers for thousands of years.
My point is that today we face a crisis not of a lack of resources as those before us would have faced. Today the labor participation rate, which I much prefer to fictitious employment rates, is at a 20 year low and continuing to drop. It is today below the rates we saw when women were en masse entering the workforce in the 70s and 80s. It is dramatically higher for those younger rather than older, predominantly because rather than train and hire new recruits to do the old jobs we either obsolete them or automate them. It is also valuable to consider that even before the rise of working class women, women were serving critical functions in society as caregivers and child rearers, that we have done a lot to professionalize and integrate into that labor force as paid work, so even today the 1:3 people over the age of 16 that are not posting working income are much less likely to be doing work without pay.
If we had this rate of labor participation a thousand, or even two hundred, years ago we would have seen mass famine and social collapse because you needed way more than that ratio of people farming just to feed everyone. We obviously do not have that problem today. Hell, we cannot even fathom an unemployment rate where would actually reasonably see a shortage of anything. Our only concerns when dealing with unemployment is how we would have millions of destitute and homeless, not due to a lack of resources, but because they lack any money to buy them with.
On the standard of living in the US, we can practically see what is happening. I don't believe it is any adjustment for climate change. We still do not give a shit. It is simply globalization, and oligarchic state control. The US did what facists and communists did to a degree with capitalism. We treated it like religion throughout the cold war, and eroded liberty and individual empowerment to hold the ideology sacred. We killed unions and sat idle while global monopolies were formed under our stead, and while our politics enabled the gross transfer of wealth from the middle to the top, and from the middle to foreign markets where the act of producing goods traveled.
On the backs of the vast majority of American wealth falling, the rich have gotten dramatically richer, billions were lifted out of extreme poverty, and nations like Korea, China, Brazil, Mexico, and India have seen the rise of dramatic middle classes in very short spans of time from the wealth we send overseas.
That is economics driving the dissolution of median wealth, not the environment. As long as our economic policy systemically ignores negative externalities like human rights and greenhouse gasses (and it always has, we just did a better job long ago at managing them through collectivism, environmental protection, and workers rights in the era before gargantuan business interests took over most western democracies through amassed wealth <and thus power>).
Actually, we have a dire shortage of non-luxury housing.
Essential to that is how you phrased it - non-luxury. The market for the (relative) rich to buy houses is so large there are over a million of unoccupied homes[1] just to sate the potential choices of the ownership class that can afford them. There is an immense glut of building supply, rather than a shortage of it, just it is all directed to where the money is.
[1]: http://www.mintpressnews.com/empty-homes-outnumber-the-homel...
China is a great extrapolated example of this with their ghost cities[2]. The economic machine of China, using the same capitalistic peak employment model of expectation, has a tremendous civil engineers corps that needs to do something. But China takes some of the worst of capitalism and mixes it with the worst of Marxism and a large dose of totalitarianism, so rather than provide modern living to hundreds of millions there are dozens of cities built without anybody living in them, because nobody can afford to buy housing within them. But the CCP kept building them to keep the workers "busy" and keep the supply chain flowing, because they kept anticipating - in the same way many deceived Americans are - that the prosperity for the masses is coming eventually.
[2]: http://priceonomics.com/surveying-the-ghost-cities-of-china/
This just reflects on my central point - capitalist economic systems do not distribute labor or create supply based on human needs, but human wants. Those without who need have nothing to give capitalism to get their needs back, but those who have to give will have all their wants provided for and more. Not to say capitalism is not the best economic organizational system we have, it just does not align with most peoples common ethics.
That said, I think you are totally wrong about this. Just look at surnames: Baker, Smith, Shoemaker, etc. The actual names of people and their families were completely centered around their employment, or more generally, what it was they did all day long. Yes, this was medieval Europe, and conditionals abound, but the same is true in many other cultures too. The surnames of Mehta or Poddar in India relate to accountants.
More generally, you mention that people needing to do something for someone else is not what humans should do. I think this is totally false. Maybe not just 1 person/baron/emperor, but I certainly think people want to be useful to other people in their community. Ok, yes, this is a gaussian curve here, like so many other things, but I would like to think that at 1 std south people still want to help others as a natural condition for mankind.
However, I do agree with you and Adam Smith that people should be mostly doing 'nothing' all day. Most of the work to feed and house people in 1st world countries is 'done.' We can easily give out the bare essentials for life, the issue is distribution, as is claimed by Minimum income folks. Though I agree with you in a mechanical sort of way, that is not the issues. The issue is emotions. Giving people free stuff is not 'fair' for most folk. And that emotional issue of fairness, something I think drives a lot of the reasons behind laws and politics, is the elephant in the room. The question quickly come out of 'why should I work if she gets free stuff all the same?' Do not underestimate the ability for people to 'game the system' either for simply the want to appear more well off than their noisy neighbors. Keeping up on the Jones's is why you see so many new iPhones in very poor areas.
Overall, I think people exist on the bell-curve between selflessness and selfishness. Most folk are 1 std north or south of the average. The real questions are where is that mean relative to Jesus and the Devil and how wide is that std relative to that?
Note: Also, time and again we are told that the standard of living here in the US is far too high for the whole planet to be able to live like us. Nearly all scientists worth listening to say that climate change is real and we have to do something about it. Weather we like it or not, the average living condition here in the US must fall to a level our planet can sustain. Possibly we are seeing this in action already.