In addition to those conceptual differences, the actual act of transferring files between remote and local on git/hg is also much faster than SVN. Entire checkouts of git repositories are usually smaller (disk size) than the equivalent SVN checkout... of one version. This means less time on the network all around.
When you send commits to a remote server, git compresses them and sends the packfile over the network. The server will refuse non-fast forward merges (the branch you're pushing is supposed to be merged by you first, locally) so it pretty much has nothing to do but uncompress the changes and apply them. Unless you have a really slow network or are sending an insane number of commits, it's not going to take longer than a few seconds.
Cloning big repositories (like emacs, with ~25 years of history) can take a while if you have a slow network, but sometimes it feels like git can actually clone a repository faster than SVN can check out a single revision.
But repos like emacs' are rare. Often, SVN repositories end up being huge because they contain the code of many unrelated projects. In Git, having such a huge repository is impractical; each project should have its own repository. There is support for submodules though, so you can make a "master" repo that contains pointers to the child repositories, if you really have to.
Can anyone say if Mercurial is significantly faster?
After going through Joel's tutorial it looks like Mercurial keeps history from branches in the root much better than either TFS or Subversion. I think I would opt for Mercurial the next time I have to set up source control.