Not self-consciousness, perhaps, but artificial instincts would be a next big thing. Imagine that your self-driving car is afraid of self-destruction and takes care of its own fitness by avoiding the holes on the tarmac.)

This, by the way, would be a major source of additional feedback for better training. Fear has been evolved for a purpose.