Except for not having direct camera and sensor access, the API isn't that limited. It just requires a little bit more thought in terms of how you lay out the UI and design how your user will interact.
I've seen some pretty impressive stuff with Glass thus far, and Glass has only been shipping for about 3 weeks. Even now only a small fraction of the 2000 developers from I/O have actually gotten their devices yet.
As Glass seem to use the same SoC as PandaBoard I suspect it should be somehow possible to build a Linux kernel with fbdev, ALSA and V4L2, then run some tiny userspace with X11 and PulseAudio servers.
Too bad the API Google released a while ago is so limited.