I switched to super-light-weight inside shoes and couldn't be happier. They work like slippers in a pinch, or like proper shoes if you pull up the heel.

https://www.amazon.com/gp/product/B089MB3NLN