Just because some people adopted the word in some narrow context doesn’t change its meaning outside of it. There’s many behaviors that are common and acceptable towards your significant other and yet completely demeaning when applied to a barista (that doesn’t happen to be your significant other). So, whatever people call each other while watching Star Wars reruns is utterly irrelevant here.
It's also possible to use it in a belittling way, but I can't think of any synonym for "woman" which couldn't, if you use the right tone of voice. That's just general sexism, and the word "babe" wouldn't be worse than "lady" with the same delivery.
If you're talking about something that's not in those two categories, could you like me to it so I can see what you're talking about?
But yes, honey and sweetheart would work just as badly. And yes, there's quite a few words that are worse. But "hey, they could do worse!" is no real argument. Sure they could.