There are limits to that of course. Aside from that, I don't think it changes the bigger pix that "everything" involved in making a non-productive capital investment is unavailable for productive investments. Can't build a rail line thru there, can't that wood and stone into an office building. All that energy and labor invested, you're never going to get a return on that.
I'm not saying a world devoid of art and fun is a good goal, but am saying that going way overboard to one extreme is going to result in blowback.
As a technicality, the whole point of modern banking is that banks can loan out the same money over, and over, and over again.
So in fact you can eat your cake and have it too.