I disagree that a decrease in productivity would make society worse as a whole. For example, the change from Serfdom to a more standard 40-hour work week certainly lowered productivity, but it also made society better.

Also productivity is complicated because humans are complicated. You may assume that, say, 50 hours of work is more productive than 40. But I doubt it - from what I've seen, it might be less productive. Even though more time is spent.

The average productivity per person today is vastly greater compared to subsistence farming times. We have much more leisure time now made possible by an increase in productivity due to machines, technology, and better division of labour. (Though I argue our current debt based money system which requires constant inflation is stealing our productivity and that's why we still feel like we're on a treadmill despite massive gains in production). This greater productivity is what allows us to consume not only more stuff, but a wider variety of stuff, and stuff of better quality than in the past.

As for 'the change from serfdom', if you're referring to the improved working conditions following the peasants revolt, remember it took the Black Death to wipe out 50% of the population in England, consumption went down, but the potential productivity of survivors would not have changed much, so landowners had to pay more for laborers to attract manpower for farming.

A decrease in productivity relative to consumption is not good, as it just causes low-supply driven inflation and makes everybody poorer. That being said, I believe the quality of productivity / consumption also need to be considered, not just aggregates. It's entirely possible a lot of productivity is just garbage products and irrational consumers are happy to consume or are tricked/bamboozled. ZIRP and excess monetary expansion facilitates these distortions, giving investors cheap money to stimulate production in dubious industries, and giving consumers cheaper credit to purchase products of dubious value.

I think we're thinking of productivity differently. Humans, themselves, are not more productive. But our output is more due to technology.

I think sometime in the future working will be completely optional. Would this be infinite productivity, or zero?

https://en.wikipedia.org/wiki/Workforce_productivity Near zero time spent working to produce infinite goods - so yes it would be infinite as per the definition.