They could've made that work and avoided the stigma they ended up facing, until adding a camera was more reasonable.
Would that featureset have been that useful? Maybe not, but then was the Google Glass we got that useful? Didn't seem so.
The only real use for Glass was a HUD + low quality camera, considering the positioning of the actual display. You don't need a camera for a HUD. Now if Glass went after augmented reality, then I'd totally agree with you.
There were some cool tech demos that exploited the camera rather impressively - facial recognition for one, but all of these are technological curiosities rather than mass-market useful. There was never a use case that seemed relevant to the everyday user where the camera was really all that useful.
I do think the camera contributed substantially to the failure of the product. More than the existence of the camera was how it was handled - no record/activity light as has been customary on many such devices, and the design of it felt viscerally like a hidden camera.
Which isn't to say Google intended to create a hidden camera, but in trying to make it blend and look like normal glasses it made the camera seem less upfront, more dishonest, and more intrusive than, say, a guy who is literally wearing a camera on his head.
Too bad it didn't have many other uses, and made being out in public kind of awkward.
The press made a big deal of the camera creeping people out, but in practice, the reaction was much more curiosity and interest than fear. I think the press just wanted something to write about, and in the absence of revolutionary abilities and interesting use cases, they wrote about its perceived flaws and did a bit of fear mongering.
Smart glasses with no way to get visual input from the environment would be a much less useful product. So, that means having a camera, and if you have a camera you can record the input from it. I'll be very surprised if anyone ever releases smart glasses with no camera.
Having said that, I'm not surprised Glass isn't doing so well. I think our culture isn't quite at the point where an always-on wearable video camera is socially acceptable. Give it a couple of years, and it probably will be.