I usually play it on a tty, partly because silly decisions, but also I can ssh into a my system that has it installed. I haven't seen a 'noticeable' slow down, but then again most of my forts barely make it to 80 dwarves.
Hmmm.... that's always been my chief complaint about DF - that it isn't text/console based ...
I don't want to run DF on my local system - I want my DF world running on a server in a rack somewhere and I connect over screen/SSH when I need to.
Is that a sane use-case with recent DF releases ?
From my experience a few releases ago, the game is completely playable in text mode, over SSH. Like others have said, it's a little slower. I think some key combinations work differently too.
I wouldn't recommend it, because it's noticeably slower than the SDL or OpenGL renderers, but you can.