Also it looks like XV did video resizing and some color mapping (https://en.wikipedia.org/wiki/X_video_extension). Today’s GPUs are doing the rendering. Filling the display buffer using primitives and textures, and outputting the display buffer are different activities that are both done by the GPU today, but it sound like XV didn’t do the first part at all, and that first part is what we’re talking about here.
X without 2D acceleration was crawlish even under my AMD Athlon.
There are certain kinds of rendering and corner cases where CPUs can have an edge, but those happen in areas like scientific visualization or high end VFX, they don’t come up often in text editor rendering engines.
You can’t use your anecdote of 1 app that might have a broken implementation to question GPUs categorically. I’ve never noticed iTerm2 using significant CPU. My iTerm2 sits at 0.0% CPU while idle. Maybe your install is busted?