Smart speakers are made to listen by design, that's their sole purpose: to listen and respond to that. Disabling the microphone takes away the core functionality and basically their purpose for being. You're left with a speaker.

For a phone the assistant is an optional side-feature and disabling it takes almost nothing away from the utility of the smartphone. You're still left with a smartphone.