Yes, X11 is very quick when you don't have compositing and special-effects layers between the client window and the display. :) For best results use an old-school, 90s style WM.
Enjoy it while it lasts; The Powers That Be (mainly Red Hat) have decided that Wayland and compositing are the future.
About the latency thing ... having recently switched back from OSX to Gentoo as my laptop's native desktop, Linux is great. Latency on all operations even in X11 are noticeably lower and more pleasant than even typing in iTerm2 on a top end Macbook Pro, and root on ZFS is awesome :)
Primary issues with console are UTF-8 support and irritations around mode setting, font resizing, and video chip (re-)initialization when booting and coming to/from X11, since my laptop (Dell XPS15 9560) has two graphics chips and they don't play perfectly nice together under many Linux kernel configurations. Angband latency, though, is awesome :)