I think the problems you described could be solved if we made tiny adjustments to our lifestyles and environments to make it much easier for automated systems to navigate and interact with it. For example make a choice of not having rugs, having QR codes in rooms, having a track on the ladder in your home, etc. The clothing and dishes could be customized to faciliate handling as well.

It seems to me that western people have become too entrenched in their familiar lifestyle and it is too hard for current technology to adapt to it as is. And so we live in a status quo where a tiny rich minority can afford to hire human servants to do their chores while middle class can neither hire human servants nor buy robotic ones.