Regards agriculture: there remain countries in which the vast majority (over 90%) of the population is engaged in agriculture. Much agricultural productivity has come through vastly increased inputs and capital, and not all countries have access to or can afford these.

The excess labor from agriculture was absorbed into other fields (so to speak). Not always painlessly -- working conditions in early Industrial Revolution factories were terrible (see Toynbee and Carlyle for descriptions, as well as, of course, Dickens). But once the kinks were worked out, we ended up with carpenters and welders and artists and writers and dog walkers.

My own view is that we're starting to run a bit far in that direction, but the coming shortage of raw materials and energy will likely address that problem for us, if not how many would prefer it be done.