There is a YC company (Wit.ai) building systems to enable this for 'internet of things' devices.

Voice makes a good interface for these devices, but I do agree that privacy is a concern.