But they still look kind of off when wearing them in public (slightly bigger than normal sunglasses, cable from one ear and other people can see the light from the screens from the side or back).

And the lack of integration is a pain - the phone has to be unlocked so is subject to random taps and swipes in my pocket.

However, an Apple-built Carplay-style projection into XReal type glasses could work very well - the question being how would you control it?

I think it is not totally necessary to have it be impossible to tell you are wearing the things. Like walking around with earbuds or portable headphone was unusual at some point (even the walkman is less than 50 years old, right). They just have to not look deeply goofy, like current VR-pressganged-into-AR headsets do.