As for trust, first, what we generally know is that any complex system is likely to have flaws. So any remote access system is likely a back door - intentionally or not. A physical off switch would be prudent (arguably switching off wireless network card and not connecting an ethernet cable is one option: but then your options are: no networking or possible vulnerability).
It's not as if any networked machine is "perfectly secure" - but it'd be nice to be able to set the scope of exposure even more easily, and granularly.
Second, one thing is trusting Intel to not intentionally install a back-door - keeping in mind that includes any "golden key" for updates. And not leaving any holes in the interface they expose (ie:getting the feature right). That in itself is a pretty high bar, imnho. But another thing is trusting Intel not to build in secret backdoors - that is; if they say, no, there's no IME in product X, then trust them on that. That is to me a much more trivial thing to trust. I know there will be bugs in the software and hardware, but I'm not all that afraid of maliciousness.
If I was working for a Chinese intelligence service the threat model might have been different.
In sum, given that there's bound to be bugs, and might be well-intentioned software upgrade support - I would prefer if it was easier to buy high-end parts without stuff like IME (this also applies to AMD btw) - or at least that it was easy to remove a jumper or toggle a dip switch to turn the stuff completely off.
[ed: finally re: server Vs laptop: there are some things that doesn't matter as much if it's exposed on a server: got my encrypted backups? Encrypted email? No problem. Got my secret key from my laptop? that is a problem]
Nearly everyone reading this on a laptop/desktop system is doing so on an Intel processor. Go ahead and attempt to enable remote access without a supported proc/chipset/bios and then come back and complain about this unexpected remote access.
There is a lot of misunderstanding about this technology in the security world which strikes me odd because it's well-documented and readily available for your own testing.