If I have to hit a button to activate the voice assistant, that removes use-cases like "my hands are full but I want to turn on the smart lights" and "I'm cooking and want a timer, but my hands are too dirty." These are the use-cases where the tool really shines because it has no competition.

Without such a use-case, the tool gets put in the back-of-mind. Sure it might be marginally easier to use than swiping and poking, but my mental model of using the phone is already swiping and poking.