The reason why less than 1% of the population can work on agriculture is because the 1% that work on agriculture are supported by highly specialized individuals that provide advanced farming equipment, weather modeling, genetically engineered seeds, financial products that help forecast demand and supply capital, etc. In a world where only 1% work, the supply chains for those things fall apart, and it then takes more than 1% of the population to feed the rest.