You don't need to rely on a server. With Simplified Payment Verification (section 8 of the bitcoin paper[1][2]) you can verify payments by querying untrusted peers, without downloading the full blockchain.

Bitcoin Wallet[3] for Android and Multibit[4] do this, which is probably why they are now the recommended clients and Electrum is not.

Edit: apparently Electrum also double-checks what the server is saying with SPV. I'm a bit fuzzy on why it still needs a server.

[1] http://bitcoin.org/bitcoin.pdf

[2] https://en.bitcoin.it/wiki/Thin_Client_Security#Header-Only_...

[3] https://play.google.com/store/apps/details?id=de.schildbach....

[4] https://multibit.org