For me it only works because I never reuse a single terminal for unrelated things. Additionally, I have my virtual desktops grid set up so that different types of tasks are always in certain places, so if the switcher shows a terminal somewhere, I know roughly what it's for even without going to look at it. So I can't "lose" a terminal.

So it's fine for X11 terminals, but not for /dev/tty2

Don't forget you can just :!some-shell-command though - not just in vim but in lots of programs even.