Personally, I wouldn't put it past either company to massively violate my trust if the incentive stars aligned just right, and I think it's silly to pretend otherwise.
The problems of putting a surveillance device in your home and carrying one in your pocket are roughly equivalent, and the rest is a debate about which huge collection of people with a wide array of beliefs and ethics is gooder
A more likely path is law enforcement's habit of putting them in the PR hell of "we could have saved Little Tiny Baby X if these companies weren't so hell bent on protecting perverts and terrorists". Once you have the device, it's only a matter of someone making a sympathetic enough public case that they need access to it.
On that front, I think Apple is taking more proactive steps to keep themselves out of the situation, I'm just not sure it's possible to avoid it completely.
At the base of the claim that Amazon is listening to what your echo can hear is the assumption that they're lying, and at the base of the claim that your iphone isn't listening to you is the assumption that Apple is not.