Yeah this annoys me too. I just had get used to the fact that if I context switch away to a different desktop, I should switch back with ctrl-left, not with cmd-tab.