You end up needing more complex optics to focus the image comfortably onto your retina at a distance that feels comfortable. An OLED display on a glasses lens with no other optics would just look like a bright blur between you and the rest of the world.