> How do you know that it's actually off?

I take your meaning, in the sense that there's no inherent reason to trust one but not the other; but I think that it's fair to say that there's a big difference between:

    Hey, wouldn't it be handy for our users if we started storing and pre-processing audio *before* hearing 'Alexa', so that we're ready to respond instantly?  Let's quietly take down the text that says that we don't do that.

(which is a plausible reasoning process somewhere down the line) on the one (Echo) hand, and

    Hey, wouldn't it be a good idea if we ignored our users' explicit election to turn off a feature?

on the other (Siri) hand.

Unless you expect random or targeted surveillance, if it generally listened and sent packets all the time back to Apple, even if you didn't tell it to, that someone would have discovered this by now.