That's not really true. It's very easy to take a picture while holding the phone in a way that makes it look as if you're just reading it.
Without Glass, who knows if you are recording or not.
The hysterics (largely driven through anti-Google rhetoric, fwiw. People like Gruber are on the front-lines of anti-Glass hate) seem naively simplistic. In this case, there are people stealing your pin codes, and they do it in bulk, using tiny, concealed video devices that you never knew were there.
And as for everyone else, any of them, and any of your environments, can be absolutely coated with surveillance devices. High capacity, very tiny cameras are literally low-single digit prices now.
Singling out Google glass users is luddism that uses it as an easy bully victim in hopes that it makes technology suddenly regress. Only technology is long past it.
The difference is that when you record someone with a non-wearable device, it's pretty obvious. In the phone example, you have to hold it up a certain way, which will immediately alert the target (as well as anyone in the vicinity) if they are paying attention.
With Glass though, who knows if you are recording or not? AFAIK there isn't a red "recording" light that comes on, right? So as far as people are concerned, you might just be casually glancing in their direction, except you aren't.