This comment reminds me of the central thesis of Red Plenty [0] - that all the mathematical or algorithmic sorcery in the world frequently fails on contact with actual human beliefs and desires. We seldom have simple enough utility functions to meaningfully optimize in real life - all of our philosophies, theologies, and other cultural debris have evolved to manage those complexities (often imperfectly).

[0] https://www.theguardian.com/books/2010/aug/08/red-plenty-fra...